Image processing method, device, electronic device, information input system and medium

Abstract

The present embodiment discloses an image processing method, device, electronic device, information input system and computer storage medium, the method comprising: acquiring at least two frames of images captured by a binocular camera, each of the at least two frames of images representing an image of an interactive object and an interactive device, the interactive device comprising a shell and at least two interactive areas made of elastic material located on the shell; determining a timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in the at least two frames of images; determining input data based on the timing of the interactive object pressing each of the at least two interactive areas, the input data representing data input to a host through the interactive device.

Description

Technical Field

[0001] The present disclosure relates to but is not limited to computer vision processing technology, and in particular to an image processing method, device, electronic device, information input system and computer storage medium. Background Art

[0002] In the related art, there are at least two implementation schemes of keyboard and mouse. The first scheme is that the keyboard and mouse are physical keyboard and mouse. When the first scheme is adopted, a lot of noise will be generated when the keyboard and mouse are pressed. The second scheme is that a virtual keyboard and mouse device with laser projection is adopted, or a virtual reality (VR) keyboard and mouse is adopted. The second scheme will bring higher cost and power consumption. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an image processing method, device, chess-playing robot, and computer storage medium.

[0004] An embodiment of the present disclosure provides an image processing method, which may include: acquiring at least two frames of images captured by a binocular camera, each of the at least two frames of images representing an image of an interactive object and an interactive device, the interactive device comprising a shell and at least two interactive areas located on the shell and made of elastic material; determining a timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in the at least two frames of images; and determining input data based on the timing of the interactive object pressing each of the at least two interactive areas, the input data representing data input to a host through the interactive device.

[0005] In some embodiments, the determining the timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in the at least two frames of images includes: performing target recognition on the interactive object and the interactive device in the at least two frames of images to determine the relative position of the interactive object and the at least two interactive areas in each frame of the at least two frames of images; determining the timing of the interactive object pressing each of the at least two interactive areas based on the timing of each frame of the at least two frames of images and the relative position of the interactive object and the at least two interactive areas in each frame of the images.

[0006] It can be seen that the embodiment of the present disclosure can determine the pressing action of the interactive object pressing each of the at least two interactive areas based on the relative position of the interactive object and the at least two interactive areas in each frame of the image, and then, combined with the timing of each frame of the at least two frames of the image, the order of the pressing actions can be determined more accurately, thereby determining the input data more accurately.

[0007] In some embodiments, the method further includes: acquiring depth information of each pixel in each of the at least two frames of images; performing target recognition on the interactive object and the interactive device in the at least two frames of images, and determining the relative position of the interactive object and the at least two interactive areas in each of the at least two frames of images, including: performing target recognition on the interactive object and the at least two interactive areas in each of the at least two frames of images, and determining the position of the interactive object and the at least two interactive areas in each of the frames of images; determining the depth information of the interactive object and the depth information of the at least two interactive areas based on the depth information of each pixel in each of the frames of images and the position of the interactive object and the at least two interactive areas in each of the frames of images; determining the three-dimensional spatial position of the interactive object and the at least two interactive areas based on the position of the interactive object and the at least two interactive areas in each of the frames of images and the depth information of the interactive object and the at least two interactive areas; determining the relative position of the interactive object and the at least two interactive areas based on the three-dimensional spatial position of the interactive object and the at least two interactive areas.

[0008] It can be seen that the embodiment of the present disclosure can more accurately determine the three-dimensional spatial position of the interactive object and the interactive area based on the depth information of each pixel point in each frame of the image, thereby more accurately determining the relative position of the interactive object and the interactive area.

[0009] In some embodiments, the determining of input data based on the timing of the interactive object pressing each of the at least two interactive areas includes: determining input information corresponding to each pressed interactive area in the at least two interactive areas and a pressing time point of each pressed interactive area based on the timing of the interactive object pressing each of the at least two interactive areas; determining input data based on the input information corresponding to each pressed interactive area and a pressing time point of each pressed interactive area.

[0010] It can be seen that the embodiments of the present disclosure can determine the input data more accurately based on the input information corresponding to the interaction area and the time point when the interaction area is pressed.

[0011] In some embodiments, the interactive device is a mouse; the method further includes: determining movement information of at least one movement of the mouse by performing target tracking on the interactive object and the mouse in the at least two frames of images, the movement information of each movement of the mouse including position change information, movement start time and movement end time; determining input data based on the timing of the interactive object pressing each of the at least two interactive areas includes: determining the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse, and the movement information of at least one movement of the mouse.

[0012] It can be seen that the embodiment of the present disclosure can determine the input data based on the movement information of at least one movement of the mouse and the timing of the interactive object pressing the various interactive areas of the mouse; because the movement information of the mouse after the interactive area of ​​the mouse is pressed can reflect the dragging operation of the mouse, the embodiment of the present disclosure can identify the dragging operation of the mouse, which is conducive to a more comprehensive implementation of the mouse inputting data to the host.

[0013] In some embodiments, the mouse is provided with a scroll wheel; the method further comprises: determining at least one scroll wheel action realized by the interactive object by performing target tracking on the interactive object and the scroll wheel of the mouse in the at least two frames of images; determining attribute data of each scroll wheel action in the at least one scroll wheel action, the attribute data comprising: a starting time point of the scroll wheel action, a duration of the scroll wheel action, and a scroll wheel speed of the scroll wheel action; accordingly, determining the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse, and movement information of at least one movement of the mouse comprises: determining the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse, movement information of at least one movement of the mouse, and attribute data of each scroll wheel action in the at least one scroll wheel action.

[0014] It can be seen that the embodiments of the present disclosure can determine the dragging event of the mouse by identifying the movement information of at least one movement of the mouse on the basis of determining the timing of the interactive object pressing the various interactive areas of the mouse; and can identify at least one scroll wheel action of the mouse by determining the attribute data of each scroll wheel action in at least one scroll wheel action, thereby more comprehensively identifying the data input into the host through the mouse.

[0015] The disclosed embodiment also provides an image processing device, which may include: an acquisition module, a first processing module and a second processing module; wherein the acquisition module is used to acquire at least two frames of images captured by a binocular camera, each of the at least two frames of images representing an image of an interactive object and an interactive device, the interactive device comprising a shell and at least two interactive areas made of elastic material located on the shell; the first processing module is used to determine the timing of the interactive object pressing each of the at least two interactive areas by tracking the interactive object and the interactive device in the at least two frames of images; the second processing module is used to determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, the input data representing data input to the host through the interactive device.

[0016] In some embodiments, the first processing module is used to determine the timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in the at least two frames of images, including: performing target recognition on the interactive object and the interactive device in the at least two frames of images, and determining the relative position of the interactive object and the at least two interactive areas in each frame of the at least two frames of images; determining the timing of the interactive object pressing each of the at least two interactive areas based on the timing of each frame of the at least two frames of images and the relative position of the interactive object and the at least two interactive areas in each frame of the image.

[0017] It can be seen that the embodiment of the present disclosure can determine the pressing action of the interactive object pressing each of the at least two interactive areas based on the relative position of the interactive object and the at least two interactive areas in each frame of the image, and then, combined with the timing of each frame of the at least two frames of the image, the order of the pressing actions can be determined more accurately, thereby determining the input data more accurately.

[0018] In some embodiments, the acquisition module is further used to acquire depth information of each pixel in each of the at least two frames of images; the first processing module is used to perform target recognition on the interactive object and the interactive device in the at least two frames of images, and determine the relative position of the interactive object and the at least two interactive areas in each of the at least two frames of images, including: performing target recognition on the interactive object and the at least two interactive areas in each of the at least two frames of images, and determining the position of the interactive object and the at least two interactive areas in each of the frames of images; determining the depth information of the interactive object and the depth information of the at least two interactive areas based on the depth information of each pixel in each of the frames of images and the position of the interactive object and the at least two interactive areas in each of the frames of images; determining the three-dimensional spatial position of the interactive object and the at least two interactive areas based on the position of the interactive object and the at least two interactive areas in each of the frames of images and the depth information of the interactive object and the at least two interactive areas; determining the relative position of the interactive object and the at least two interactive areas based on the three-dimensional spatial position of the interactive object and the at least two interactive areas.

[0019] It can be seen that the embodiment of the present disclosure can more accurately determine the three-dimensional spatial position of the interactive object and the interactive area based on the depth information of each pixel point in each frame of the image, thereby more accurately determining the relative position of the interactive object and the interactive area.

[0020] In some embodiments, the second processing module is used to determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, including: determining input information corresponding to each pressed interactive area in the at least two interactive areas and a pressing time point of each pressed interactive area based on the timing of the interactive object pressing each of the at least two interactive areas; determining input data based on the input information corresponding to each pressed interactive area and a pressing time point of each pressed interactive area.

[0021] It can be seen that the embodiments of the present disclosure can determine the input data more accurately based on the input information corresponding to the interaction area and the time point when the interaction area is pressed.

[0022] In some embodiments, the interactive device is a mouse; the first processing module is further used to determine the movement information of at least one movement of the mouse by tracking the interactive object and the mouse in the at least two frames of images, and the movement information of each movement of the mouse includes position change information, movement start time and movement end time; the second processing module is used to determine the input data based on the timing of the interactive object pressing each of the at least two interactive areas, including: determining the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse, and the movement information of at least one movement of the mouse.

[0023] It can be seen that the embodiment of the present disclosure can determine the input data based on the movement information of at least one movement of the mouse and the timing of the interactive object pressing the various interactive areas of the mouse; because the movement information of the mouse after the interactive area of ​​the mouse is pressed can reflect the dragging operation of the mouse, the embodiment of the present disclosure can identify the dragging operation of the mouse, which is conducive to a more comprehensive implementation of the mouse inputting data to the host.

[0024] In some embodiments, the mouse is provided with a scroll wheel; the first processing module is further used to determine at least one scroll wheel action achieved by the interactive object by tracking the interactive object and the scroll wheel of the mouse in the at least two frames of images; determine attribute data of each scroll wheel action in the at least one scroll wheel action, and the attribute data include: the starting time point of the scroll wheel action, the duration of the scroll wheel action, and the scroll wheel speed of the scroll wheel action; accordingly, the second processing module is used to determine the input data based on the timing of the interactive object pressing the interactive areas of the mouse, and the movement information of at least one movement of the mouse, including: determining the input data based on the timing of the interactive object pressing the interactive areas of the mouse, the movement information of at least one movement of the mouse, and the attribute data of each scroll wheel action in the at least one scroll wheel action.

[0025] It can be seen that the embodiments of the present disclosure can determine the dragging event of the mouse by identifying the movement information of at least one movement of the mouse on the basis of determining the timing of the interactive object pressing the various interactive areas of the mouse; and can identify at least one scroll wheel action of the mouse by determining the attribute data of each scroll wheel action in at least one scroll wheel action, thereby more comprehensively identifying the data input into the host through the mouse.

[0026] An embodiment of the present disclosure further provides an electronic device, which includes a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to run the computer program to execute any one of the above-mentioned image processing methods.

[0027] The embodiment of the present disclosure also provides an information input system, which is communicatively connected to a host, and includes a binocular camera and at least one interactive device. The information input system also includes any one of the above-mentioned image processing devices or any one of the above-mentioned electronic devices.

[0028] The embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned image processing methods is implemented.

[0029] It can be seen that the embodiments of the present disclosure do not need to adopt traditional keyboard and mouse devices, nor do they need to adopt the laser projected virtual keyboard and VR keyboard in the related art. Instead, the action of the interactive object pressing the interactive area can be identified through target tracking technology, thereby determining the data input by the interactive device to the host. On the one hand, when no springs, switch components, and circuit array structures are set inside the interactive device, the target tracking technology can also be used to identify the data input by the interactive device to the host. Therefore, the noise emitted by the interactive device is reduced to a certain extent, which is conducive to silent pressing. On the other hand, the embodiments of the present disclosure do not need to use laser projection equipment or VR equipment, but can collect images through a binocular camera with lower cost and power consumption. Compared with the laser projected virtual keyboard and mouse device or VR keyboard and mouse in the related art, the cost and power consumption can be reduced.

[0030] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0032] Figure 1 is a flow chart of an image processing method according to an embodiment of the present disclosure;

[0033] Figure 2 is a flowchart of another image processing method according to an embodiment of the present disclosure;

[0034] Figure 3 A schematic diagram of a process for determining relative positions of an interactive object and at least two interactive areas in an embodiment of the present disclosure;

[0035] Figure 4 is a flowchart of another image processing method according to an embodiment of the present disclosure;

[0036] Figure 5 is a flowchart of another image processing method according to an embodiment of the present disclosure;

[0037] Figure 6 is a flowchart of another image processing method according to an embodiment of the present disclosure;

[0038] Figure 7 A schematic diagram of the spatial distribution of the information input system in an embodiment of the present disclosure;

[0039] Figure 8 A schematic diagram of the circuit connection between the information input system and the host in the embodiment of the present disclosure;

[0040] Fig. 9 A flowchart of a method for inputting data to a host in an embodiment of the present disclosure;

[0041] Fig.10 is a structural schematic diagram of a data processing device according to an embodiment of the present disclosure;

[0042] Fig.11 A schematic diagram of a hardware entity of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0044] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0047] In order to better understand the technical solution of the embodiment of the present disclosure, the keyboard and mouse device in the related art is first described below.

[0048] In the related technology, most companies in office buildings adopt the mode of centralized office for employees. Except for a few high-ranking employees who have independent offices, most employees have desks next to each other. Therefore, the most common sounds heard in the office are the sounds of keyboard tapping and mouse clicking. Although this noise has been improved with the improvement of manufacturing technology, it is still impossible to eliminate it. This makes people who often need quiet and need to think quietly feel annoyed, and to some extent reduces work efficiency. In addition to the mute problem, after the keyboard and mouse are used for a long time, the mechanical keyboard and mouse are easy to hide dirt and are not easy to clean, which is not conducive to improving the office environment.

[0049] There are other ways to implement keyboard and mouse, such as laser projection virtual keyboard, or VR virtual keyboard. However, this solution also has obvious disadvantages, that is, either it changes people's years of habitual usage, or it has no feel at all. In addition, laser projection needs to emit light source, which is not good for the human eye and greatly increases power consumption. The entire system architecture of VR virtual keyboard is complex, the cost is high, and the application range is narrow. In other words, the cost and power consumption of this solution are also high. Therefore, users are not very accepting of it.

[0050] In view of the above technical problems, the embodiment of the present disclosure proposes an image processing method, which can be applied to an electronic device, and the electronic device may include a processor. The above processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0051] Figure 1 is a flowchart of an image processing method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process may include:

[0052] Step 101: Acquire at least two frames of images captured by a binocular camera, each of the at least two frames of images represents an image of an interactive object and an interactive device, and the interactive device includes a shell and at least two interactive areas made of elastic material located on the shell.

[0053] In the disclosed embodiment, the binocular camera is a device that uses binocular stereo vision technology. Based on the images captured by the binocular camera, the depth information of each pixel in the image can be determined, thereby determining the three-dimensional spatial position corresponding to each pixel in the image.

[0054] The binocular camera can capture images in real time, and the at least two frames of images captured by the binocular camera may be images captured continuously or may not be images captured continuously. In practical applications, the capture time of each frame of image captured by the binocular camera can be determined, thereby determining the timing of the at least two frames of images captured by the binocular camera.

[0055] The binocular camera and the above-mentioned electronic device can form a communication connection so as to send the image captured by the binocular camera to the electronic device; illustratively, the binocular camera can be connected to the electronic device via a universal serial bus (USB) or other types of communication buses.

[0056] In the embodiments of the present disclosure, the interactive object may be an object such as a human finger for operating the interactive device, and the interactive device may be a simulated physical device such as a keyboard and a mouse; the shell in the interactive device may be located at the bottom of the interactive device, for supporting the interactive area; the interactive area may be a simulated physical button, etc.

[0057] Since the interactive area of ​​the interactive device is made of elastic material, there is no need to set devices such as springs inside the interactive device. After pressing the interactive area and then releasing it, the interactive area will automatically return to its original state.

[0058] Here, the interactive device is different from a conventional mechanical keyboard and mouse device. The interactive device may only include a shell and an interactive area, and there is no need to set springs, switch components, and circuit array structures inside.

[0059] Step 102: determining the timing of the interactive object pressing each of the at least two interactive areas by tracking the interactive object and the interactive device in at least two frames of images.

[0060] In the embodiment of the present disclosure, target tracking can be performed through the interactive object and the interactive device in at least two frames of images to determine the action of the interactive object pressing each of the at least two interactive areas; and the sequence relationship of the actions of the interactive object pressing each of the at least two interactive areas can be derived based on the time sequence of the at least two frames of images.

[0061] Step 103: Determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, where the input data represents data input to the host through the interactive device.

[0062] In the disclosed embodiment, the host may be a terminal device such as a personal computer (PC), a mobile device, etc. The electronic device may form a communication connection with the host. In one implementation, in order to achieve reliable data transmission between the electronic device and the host, the electronic device may exchange data with the host via USB 3.0.

[0063] Here, based on the timing of the interactive object pressing each of the at least two interactive areas, the time sequence of the action of the interactive object pressing each of the at least two interactive areas can be determined, so that according to the time sequence of the action of the interactive object pressing each of the at least two interactive areas, the action of the interactive object pressing each of the at least two interactive areas is recorded in sequence, and the input data can be determined according to the actions recorded in sequence. In one example, the interactive device is a simulated physical keyboard, and when a person presses the g key, the u key, and the o key of the simulated physical keyboard in sequence, the input data can be considered to be "guo"; in another example, the interactive device is a simulated physical mouse, and when a person presses the left button of the simulated physical mouse twice in sequence, and the time interval between the two pressings of the left button of the simulated physical mouse is less than 0.5 seconds, the input data can be considered to be a double-click event of the mouse.

[0064] It can be seen that the embodiments of the present disclosure do not need to adopt traditional keyboard and mouse devices, nor do they need to adopt the virtual keyboard and VR keyboard of laser projection in the related art. Instead, the action of the interactive object pressing the interactive area can be identified through target tracking technology, thereby determining the data input by the interactive device to the host. On the one hand, in the case that no springs, switch components, and circuit array structures are set inside the interactive device, the noise emitted by the interactive device is reduced to a certain extent, which is conducive to silent pressing. On the other hand, the embodiments of the present disclosure do not need to use laser projection equipment or VR equipment, but can collect images through a binocular camera with low cost and power consumption, which can reduce cost and power consumption compared with the virtual keyboard and mouse device or VR keyboard and mouse of laser projection in the related art.

[0065] In some embodiments of the present disclosure, Figure 2 , the process of another image processing method of the embodiment of the present disclosure may include:

[0066] Step 201: Acquire at least two frames of images captured by a binocular camera, where each of the at least two frames of images represents an image of an interactive object and an interactive device.

[0067] Here, the above step 201 corresponds to the above step 101, and the specific implementation of the above step 101 may be referred to during implementation.

[0068] Step 202: performing target recognition on the interactive object and the interactive device in at least two frames of images, and determining the relative position of the interactive object and at least two interactive areas in each of the at least two frames of images.

[0069] In the embodiments of the present disclosure, target recognition can be performed on the interactive objects and interactive devices in each frame of image, respectively, to determine the positions and postures of the interactive objects and interactive devices in each frame of image; based on the positions and postures of the interactive objects and interactive devices in each frame of image, the relative positions of the interactive objects and at least two interactive areas in each frame of image can be determined; it can be understood that based on the relative positions of the interactive objects and at least two interactive areas in each frame of image, the type of interactive area pressed by the interactive object can be identified, for example, the type of key pressed by a human hand can be identified.

[0070] Step 203: Determine the timing of the interactive object pressing each of the at least two interactive areas based on the timing of each of the at least two image frames and the relative position of the interactive object and the at least two interactive areas in each image frame.

[0071] In an embodiment of the present disclosure, after determining the relative position of the interactive object and the at least two interactive areas in each frame of the image, combined with the timing of each frame in the at least two frames of the image, the pressing areas pressed successively by the interactive object can be determined, for example, the keys pressed successively by a person's hand can be determined.

[0072] In some embodiments, the posture of the interactive object and the posture of at least two interactive areas in each frame can also be determined by performing target recognition on the interactive object and the interactive device in each frame image. Thus, based on the relative position of the interactive object and the at least two interactive areas in each frame image, and the posture of the interactive object and the posture of the at least two interactive areas, the relative posture of the interactive object and the at least two interactive areas in each frame image can be determined; based on the timing of each frame image in at least two frames of images, and the relative posture of the interactive object and the at least two interactive areas in each frame image, the timing of the interactive object pressing the interactive area in the at least two interactive areas can be determined.

[0073] Step 204: Determine input data based on the timing of the interactive object pressing the interactive area of ​​at least two interactive areas.

[0074] Here, the above step 204 corresponds to the above step 103, and the specific implementation of the above step 103 may be referred to during implementation.

[0075] It can be seen that the embodiment of the present disclosure can determine the pressing action of the interactive object pressing each of the at least two interactive areas based on the relative position of the interactive object and the at least two interactive areas in each frame of the image, and then, combined with the timing of each frame of the at least two frames of the image, the order of the pressing actions can be determined more accurately, thereby determining the input data more accurately.

[0076] In some embodiments of the present disclosure, after acquiring at least two frames of images captured by a binocular camera, depth information of each pixel in each of the at least two frames of images may be acquired.

[0077] Accordingly, refer to Figure 3 , the implementation process of step 202 may include:

[0078] Step 2021: Perform target recognition on the interactive object and at least two interactive areas in each frame of the image, and determine the positions of the interactive object and the at least two interactive areas in each frame of the image.

[0079] Step 2022: Determine the depth information of the interactive object and the depth information of the at least two interactive areas based on the depth information of each pixel in each frame of the image and the positions of the interactive object and the at least two interactive areas in each frame of the image.

[0080] Here, based on the position of the interactive object in each frame of image, the depth information of the interactive object in each frame of image can be found out from the depth information of each pixel in each frame of image; similarly, based on the position of each interactive area in each frame of image, the depth information of each pixel in each frame of image can be found out from the depth information of each interactive area in each frame of image.

[0081] Step 2023: Determine the three-dimensional spatial position of the interactive object and the at least two interactive areas based on the positions of the interactive object and the at least two interactive areas in each frame of the image and the depth information of the interactive object and the at least two interactive areas.

[0082] In the embodiments of the present disclosure, the depth information of the interactive object can represent the distance between the interactive object and the depth camera. Thus, based on the distance between the interactive object and the depth camera, and the position of the interactive object in each frame image, the three-dimensional coordinates of the interactive object can be determined in the world coordinate system, that is, the three-dimensional spatial position of the interactive object is determined; in some embodiments, based on the distance between the interactive object and the depth camera, and the position of the interactive object in each frame image, the two-dimensional coordinate position of the interactive object in each frame image can be converted through the coordinate system to obtain the three-dimensional coordinates of the interactive object in the world coordinate system.

[0083] Similarly, the depth information of the interaction area can represent the distance between the interaction area and the depth camera. Thus, based on the distance between the interaction area and the depth camera and the position of the interaction area in each frame image, the three-dimensional coordinates of the interaction area can be determined in the world coordinate system, that is, the three-dimensional spatial position of the interaction area can be determined.

[0084] Step 2024: Determine the relative position of the interactive object and the at least two interactive areas based on the three-dimensional spatial position of the interactive object and the at least two interactive areas.

[0085] Here, the relative position between the interactive object and the at least two interactive areas indicates: the relative relationship between the three-dimensional spatial positions of the interactive object and the at least two interactive areas.

[0086] It can be seen that the embodiment of the present disclosure can more accurately determine the three-dimensional spatial position of the interactive object and the interactive area based on the depth information of each pixel point in each frame of the image, thereby more accurately determining the relative position of the interactive object and the interactive area.

[0087] In some embodiments of the present disclosure, Figure 4 , another image processing method of the embodiment of the present disclosure may include:

[0088] Step 401: Acquire at least two frames of images captured by a binocular camera, where each of the at least two frames of images represents an image of an interactive object and an interactive device.

[0089] Step 402: Determine the timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in at least two frames of images.

[0090] Here, the above steps 401 to 402 correspond to the above steps 101 to 102 respectively, and the specific implementation methods of the above steps 101 to 102 may be referred to during implementation.

[0091] Step 403: Based on the timing of the interactive object pressing each of the at least two interactive areas, determine the input information corresponding to each pressed interactive area in the at least two interactive areas and the pressing time point of each pressed interactive area.

[0092] Step 404: Determine input data based on the input information corresponding to each pressed interaction area and the pressing time point of each pressed interaction area.

[0093] Exemplarily, the interaction area is a simulated physical key of a simulated physical keyboard, the time point when the simulated physical key a is pressed is time point 1, the time point when the simulated physical key b is pressed is time point 2, time point 2 is later than time point 1, the input information corresponding to the simulated physical key a is "a", the input information corresponding to the simulated physical key b is "b", and the input data is "ab".

[0094] It can be seen that the embodiments of the present disclosure can determine the input data more accurately based on the input information corresponding to the interaction area and the time point when the interaction area is pressed.

[0095] In some embodiments of the present disclosure, the above-mentioned interactive device is a mouse; the movement information of at least one movement of the mouse can be determined by tracking the interactive object and the mouse in at least two frames of images, and the movement information of each movement of the mouse includes position change information, movement start time and movement end time.

[0096] Here, the position change information may include the position of the mouse before this movement, and the position of the mouse after this movement; combined with the start time and end time of the movement, the spatiotemporal change information caused by this movement of the mouse can be determined, that is, the movement information of this movement of the mouse can reflect the spatiotemporal change information of the mouse.

[0097] Accordingly, refer to Figure 5 , the process of another image processing method of the embodiment of the present disclosure may include:

[0098] Step 501: Acquire at least two frames of images captured by a binocular camera, where each of the at least two frames of images represents an image of an interactive object and an interactive device.

[0099] Step 502: Determine the timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in at least two frames of images.

[0100] Here, the above steps 501 to 502 correspond to the above steps 101 to 102 respectively, and the specific implementation methods of the above steps 101 to 102 may be referred to during implementation.

[0101] Step 503: Determine input data based on the timing of the interacting object pressing each interactive area of ​​the mouse and movement information of at least one movement of the mouse.

[0102] Here, the interaction area of ​​the mouse may include simulated physical buttons such as the left button and right button of the mouse. Combined with the timing of pressing the interaction area of ​​the mouse based on the interactive object, and the movement information of at least one movement of the mouse, the operation of dragging the information on the screen using the mouse can be determined, and the operation of dragging the information on the screen using the mouse can be determined as input data.

[0103] Of course, in other embodiments, the movement information of the cursor corresponding to the mouse can be determined based on the movement information of at least one movement of the mouse, thereby determining the movement information of the cursor corresponding to the mouse as input data.

[0104] It can be seen that the embodiment of the present disclosure can determine the input data based on the movement information of at least one movement of the mouse and the timing of the interactive object pressing the various interactive areas of the mouse; because the movement information of the mouse after the interactive area of ​​the mouse is pressed can reflect the dragging operation of the mouse, the embodiment of the present disclosure can identify the dragging operation of the mouse, which is conducive to a more comprehensive implementation of the mouse inputting data to the host.

[0105] In some embodiments of the present disclosure, a mouse is further provided with a scroll wheel. By performing target tracking on the interactive object and the scroll wheel of the mouse in at least two frames of images, at least one scroll wheel action realized by the interactive object is determined; and attribute data of each scroll wheel action in the at least one scroll wheel action is determined, the attribute data including: the start time point of the scroll wheel action, the duration of the scroll wheel action, and the scroll wheel speed of the scroll wheel action.

[0106] Here, the scroll wheel provided on the mouse may be a simulated physical scroll wheel, which may be consistent with the scroll wheel of a traditional mouse, or may be made of elastic materials such as silicone.

[0107] Combined with the above-mentioned records, when the interactive object is a human hand, target tracking is performed through the interactive object and the mouse wheel in at least two frames of images, and the attribute data of each wheel movement can be determined, that is, each wheel movement performed by the human hand can be identified.

[0108] Accordingly, refer to Figure 6 , the process of another image processing method of the embodiment of the present disclosure may include:

[0109] Step 601: Acquire at least two frames of images captured by a binocular camera, where each of the at least two frames of images represents an image of an interactive object and an interactive device.

[0110] Step 602: Determine the timing of the interactive object pressing each of the at least two interactive areas by performing target tracking on the interactive object and the interactive device in at least two frames of images.

[0111] Here, the above steps 601 to 602 correspond to the above steps 101 to 102 respectively, and the specific implementation methods of the above steps 101 to 102 may be referred to during implementation.

[0112] Step 603: Determine input data based on the timing of the interactive object pressing each interactive area of ​​the mouse, movement information of at least one movement of the mouse, and attribute data of each roller movement in at least one roller movement.

[0113] It can be seen that the embodiment of the present disclosure can determine the drag event of the mouse by identifying the movement information of at least one movement of the mouse on the basis of determining the timing of the interactive object pressing the interactive area of ​​the mouse; and can identify at least one scroll wheel action of the mouse by determining the attribute data of each scroll wheel action in at least one scroll wheel action, thereby more comprehensively identifying the data input into the host through the mouse.

[0114] Based on the contents recorded in the above embodiments, the embodiments of the present disclosure can provide a solution for an intelligent silent keyboard and mouse device based on machine vision. The embodiments of the present disclosure do not need to use a traditional mechanical keyboard and mouse, but can design a set of almost completely silent keyboard and mouse devices, which is conducive to providing a quiet working environment; the interactive device in the embodiments of the present disclosure can be made of a simulated silicone entity, and the appearance can be consistent with the traditional keyboard and mouse device, so that users can use the interactive device of the embodiments of the present disclosure to input data without changing their usage habits.

[0115] The image processing method described above is exemplified below through an application scenario embodiment. In this application scenario embodiment, data can be input into the host through an information input system, and the information input system may include an artificial intelligence (AI) algorithm processing board, a three-dimensional (3D) binocular camera, a simulated physical keyboard, and a simulated physical mouse, wherein the AI ​​algorithm processing board is the above-mentioned electronic device, and the AI ​​algorithm processing board has a powerful image data processing capability; the 3D binocular camera is a depth camera, and the simulated physical keyboard and the simulated physical mouse are both interactive devices. The communication interface of the interactive device can be the same as the communication interface of a traditional keyboard and mouse device.

[0116] Figure 7 This is a schematic diagram of the spatial distribution of the information input system in the embodiment of the present disclosure, referring to Figure 7 , the 3D binocular camera 701 can be fixed by a bracket 702, and the bracket 702 can be manually leveled through structural design to facilitate transportation through cargo packaging; the shooting range 703 of the 3D binocular camera 701 (shooting range 703 is Figure 7The range within the dotted line in the middle covers the usage range of the simulated physical keyboard 704 and the usage range of the simulated physical mouse 705. When the shooting range 703 of the 3D binocular camera 701 covers the usage range of the simulated physical keyboard 704 and the usage range of the simulated physical mouse 705, the shooting of the 3D binocular camera 701 cannot be manually adjusted to prevent the simulated physical keyboard 704 and the simulated physical mouse 705 from deviating from the shooting range of the 3D binocular camera 701.

[0117] The shooting range 703 of the 3D binocular camera 701 is an elliptical area, and the shooting range 703 can cover the entire area of ​​the simulated physical keyboard 704, the area where the left hand can be when typing on the keyboard, the area where the right hand can be when typing on the keyboard, and the area where the mouse can be. The 3D binocular camera 701 can use a high-definition starlight-level image sensor to collect images, so that a clearer interactive device image can be collected even in a weak light environment.

[0118] In some embodiments, reference Figure 7 , the AI ​​algorithm processing board can be located below the simulated physical keyboard 704; the simulated physical keyboard 704 can include a shell and simulated physical keys made of silicone, and the simulated physical mouse 705 can include a shell and simulated physical keys made of silicone, and the hardness of the silicone is moderate, which is convenient for user operation. The simulated physical keyboard 704 can be cleaned after disassembly, and the simulated physical keys in the simulated physical keyboard 704 are protruding keys and have silk screen printing. The simulated physical keys in the simulated physical keyboard 704 have a certain travel and pressing feel. In addition to the left and right keys, the simulated physical mouse 705 is also provided with a scroll wheel to facilitate compatibility with people's habitual actions.

[0119] In some embodiments, the simulated physical keyboard 704 and the simulated physical mouse 705 do not have other mechanical parts, circuit array structures, and batteries inside, and because the simulated physical keys in the simulated physical keyboard 704 and the simulated physical mouse 705 can be made of elastic silicone, the sound produced by pressing the simulated physical keyboard 704 and the simulated physical mouse 705 is small and almost silent. The size and shape of the simulated physical keyboard 704 can be consistent with that of a traditional keyboard, and the size and shape of the simulated physical mouse 705 can be consistent with that of a traditional mouse.

[0120] Figure 8 FIG. 1 is a schematic diagram of a circuit connection between an information input system and a host in an embodiment of the present disclosure. Figure 8As shown, the 3D binocular camera 701 can form a communication connection with the AI ​​algorithm processing board 801, and the AI ​​algorithm processing board 801 can form a communication connection with the host 803 through a USB cable 802. For example, the communication protocol adopted by the USB cable 802 is the USB3.0 protocol, and the AI ​​algorithm processing board 801 can be powered through the USB3.0 interface, so that the AI ​​algorithm processing board 801 can work at a sufficiently large power. In some embodiments, the 3D binocular camera 701 can also form a communication connection with the host 803.

[0121] The present disclosure provides a method for inputting data to a host, which can be applied to the AI ​​algorithm processing board 801. Fig. 9 FIG. 1 is a flow chart of a method for inputting data to a host in an embodiment of the present disclosure. Fig. 9 As shown, the process may include:

[0122] Step 901: Determine whether the self-check is successful; if not, execute step 902; if yes, execute step 903.

[0123] Here, it can be determined whether the image captured by the 3D binocular camera 701 is received. If the image captured by the 3D binocular camera 701 is received, it means that the self-check is successful; if the image captured by the 3D binocular camera 701 is not received, it means that the self-check fails.

[0124] Step 902: Determine that the driver mounting of the interactive device fails.

[0125] Step 903: Determine whether the driver of the interactive device is mounted successfully, and then execute step 904.

[0126] Here, the interactive device includes a simulated physical keyboard and a simulated physical mouse. After determining that the driver of the interactive device is successfully mounted, the input information corresponding to the interactive device can be received. Otherwise, it is not necessary to receive the input information corresponding to the interactive device.

[0127] Step 904: Determine whether the interactive device can be identified from the image captured by the 3D binocular camera. If not, re-execute step 904; if yes, execute step 905.

[0128] Step 905: Determine the relative position of the human hand and the interactive device in the three-dimensional space by performing target recognition on the human hand and the interactive device in the image.

[0129] Here, after the first frame image, the target recognition can be performed on the interactive device in the first frame image to determine the initial position of the interactive device in the three-dimensional space; here, the dual-camera triangle principle can be used to determine the initial position of the initial position of the interactive device; that is, firstly, the depth information of the interactive device in the first frame image can be determined based on the dual-camera triangle principle, and the depth information of the interactive device can reflect the distance between the interactive device and the 3D binocular camera, so that based on the distance between the interactive device and the 3D binocular camera, and the position of the interactive device in each frame image, the three-dimensional coordinates of the interactive device can be determined in the world coordinate system, that is, the initial position of the interactive device in the three-dimensional space is determined. Exemplarily, the initial position of the simulated physical mouse and the simulated physical keyboard can be determined. Similarly, the initial position of each area in the simulated physical mouse and the initial position of each area in the simulated physical keyboard can also be determined, and each area in the simulated physical mouse can include a scroll wheel and a simulated physical button.

[0130] Similarly, for non-first frame images, target recognition can be performed on human hands, interactive devices, and each interactive area to determine the positions of human hands, interactive devices, and each interactive area in three-dimensional space. In this way, the position change information of human hands, interactive devices, and each interactive area can be determined by analyzing at least two frames of images.

[0131] Step 906: Determine whether there is an interaction action between the human hand and the interactive device. If not, re-execute step 906; if yes, execute step 907.

[0132] Here, the interactive action may include a human hand pressing an emulated physical keyboard or an emulated physical mouse, a human hand rolling a wheel of the emulated physical mouse, and a human hand moving the emulated physical mouse.

[0133] In practical applications, whether there is an interaction action between the human hand and the interactive device can be determined based on the position change information of the human hand, the interactive device, and each interactive area.

[0134] Step 907: Generate input data based on the timing of each interaction action between the human hand and the interaction device.

[0135] Here, based on the timing of each interaction action between the human hand and the interaction device, the input information corresponding to each interaction action and the time point of each interaction action can be determined; based on the input information corresponding to each interaction action and the time point of each interaction action, input data can be generated.

[0136] Step 908: Send input data to the host.

[0137] In order to solve the problems of non-silence and harboring dirt in traditional keyboard and mouse devices, as well as the problem of poor operating feel in virtual keyboards, the embodiments of the present disclosure utilize a 3D binocular camera to adopt the image of an interactive device, and identify the interactive device in the image and track human hands to determine the input actions of the human hands, thereby simulating the input of a traditional keyboard and mouse device; that is, the embodiments of the present disclosure can utilize an AI algorithm processing board to realize target recognition and target tracking of the image, thereby determining the data input by the interactive device to the host through analysis of the image.

[0138] The embodiments of the present disclosure can be applied to various scenarios where data needs to be input into a host. For example, in an office environment or a home environment, an information input system using the embodiments of the present disclosure can determine the input data by analyzing the image; devices such as the AI ​​algorithm processing board of the embodiments of the present disclosure do not need to be charged and can work without batteries; silent operation of interactive devices can be achieved, and when the interactive devices are used for a long time, they can be cleaned with water after disassembly, which is very convenient; the material of the interactive devices is special silicone, which is comfortable and good to the touch.

[0139] Based on the foregoing embodiments, the embodiments of the present disclosure provide an image processing device, wherein each unit included in the device and each module included in each unit can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; during the implementation process, the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0140] Fig.10 Schematic diagram of the structure of the data processing device of the embodiment of the present disclosure. Fig.10 As shown, the device includes an acquisition module 1001, a first processing module 1002 and a second processing module 1003; wherein,

[0141] An acquisition module 1001 is used to acquire at least two frames of images captured by a binocular camera, wherein each of the at least two frames of images represents an image of an interactive object and an interactive device, wherein the interactive device includes a shell and at least two interactive areas made of elastic material and located on the shell;

[0142] A first processing module 1002 is configured to determine a timing of the interactive object pressing each of the at least two interactive areas by tracking the interactive object and the interactive device in the at least two frames of images;

[0143] The second processing module 1003 is used to determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, where the input data represents data input to the host through the interactive device.

[0144] It can be seen that the embodiments of the present disclosure do not need to adopt traditional keyboard and mouse devices, nor do they need to adopt the laser projected virtual keyboard and VR keyboard in the related art. Instead, the action of the interactive object pressing the interactive area can be identified through target tracking technology, thereby determining the data input by the interactive device to the host. On the one hand, when no springs, switch components, and circuit array structures are set inside the interactive device, the target tracking technology can also be used to identify the data input by the interactive device to the host. Therefore, the noise emitted by the interactive device is reduced to a certain extent, which is conducive to silent pressing. On the other hand, the embodiments of the present disclosure do not need to use laser projection equipment or VR equipment, but can collect images through a binocular camera with lower cost and power consumption. Compared with the laser projected virtual keyboard and mouse device or VR keyboard and mouse in the related art, the cost and power consumption can be reduced.

[0145] In some embodiments, the first processing module 1002 is configured to determine the timing of the interactive object pressing each of the at least two interactive areas by tracking the interactive object and the interactive device in the at least two frames of images, including:

[0146] Performing target recognition on the interactive object and the interactive device in the at least two frames of images, and determining a relative position between the interactive object and the at least two interactive areas in each of the at least two frames of images;

[0147] Based on the timing of each frame image in the at least two frames of images and the relative position of the interactive object and the at least two interactive areas in each frame image, the timing of the interactive object pressing each interactive area in the at least two interactive areas is determined.

[0148] It can be seen that the embodiment of the present disclosure can determine the pressing action of the interactive object pressing each of the at least two interactive areas based on the relative position of the interactive object and the at least two interactive areas in each frame of the image, and then, combined with the timing of each frame of the at least two frames of images, the order of the pressing actions can be more accurately determined, thereby more accurately determining the input data.

[0149] In some embodiments, the acquisition module 1001 is further used to acquire depth information of each pixel in each frame of the at least two frames of images;

[0150] The first processing module 1002 is used to perform target recognition on the interactive object and the interactive device in the at least two frames of images, and determine the relative position of the interactive object and the at least two interactive areas in each of the at least two frames of images, including:

[0151] Performing target recognition on the interactive object and the at least two interactive areas in each frame of the image, and determining positions of the interactive object and the at least two interactive areas in each frame of the image;

[0152] Determine the depth information of the interactive object and the depth information of the at least two interactive areas based on the depth information of each pixel in each frame of the image and the positions of the interactive object and the at least two interactive areas in each frame of the image;

[0153] Determine the three-dimensional spatial position of the interactive object and the at least two interactive areas based on the positions of the interactive object and the at least two interactive areas in each frame of the image and the depth information of the interactive object and the at least two interactive areas;

[0154] Based on the three-dimensional spatial positions of the interactive object and the at least two interactive areas, a relative position of the interactive object and the at least two interactive areas is determined.

[0155] It can be seen that the embodiment of the present disclosure can more accurately determine the three-dimensional spatial position of the interactive object and the interactive area based on the depth information of each pixel point in each frame of the image, thereby more accurately determining the relative position of the interactive object and the interactive area.

[0156] In some embodiments, the second processing module 1003 is used to determine the input data based on the timing of the interactive object pressing each of the at least two interactive areas, including:

[0157] Determining input information corresponding to each pressed interaction area of ​​the at least two interaction areas and a pressing time point of each pressed interaction area based on a timing of the interaction object pressing each interaction area of ​​the at least two interaction areas;

[0158] Input data is determined based on the input information corresponding to each pressed interaction area and the pressing time point of each pressed interaction area.

[0159] It can be seen that the embodiments of the present disclosure can determine the input data more accurately based on the input information corresponding to the interaction area and the time point when the interaction area is pressed.

[0160] In some embodiments, the interactive device is a mouse; the first processing module 1002 is further used to determine movement information of at least one movement of the mouse by tracking the interactive object and the mouse in the at least two frames of images, wherein the movement information of each movement of the mouse includes position change information, movement start time, and movement end time;

[0161] The second processing module 1003 is used to determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, including:

[0162] The input data is determined based on a timing of the interactive object pressing an interactive area of ​​the mouse and movement information of at least one movement of the mouse.

[0163] It can be seen that the embodiment of the present disclosure can determine the input data based on the movement information of at least one movement of the mouse and the timing of the interactive object pressing the various interactive areas of the mouse; because the movement information of the mouse after the interactive area of ​​the mouse is pressed can reflect the dragging operation of the mouse, the embodiment of the present disclosure can identify the dragging operation of the mouse, which is conducive to a more comprehensive implementation of the mouse inputting data to the host.

[0164] In some embodiments, the mouse is provided with a scroll wheel; the first processing module 1002 is further used to determine at least one scroll wheel action realized by the interactive object by tracking the interactive object and the scroll wheel of the mouse in the at least two frames of images; determine attribute data of each scroll wheel action in the at least one scroll wheel action, the attribute data including: a start time point of the scroll wheel action, a duration of the scroll wheel action, and a scroll wheel speed of the scroll wheel action;

[0165] Accordingly, the second processing module 1003 is used to determine the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse and the movement information of at least one movement of the mouse, including:

[0166] The input data is determined based on the timing of the interactive object pressing each interactive area of ​​the mouse, the movement information of at least one movement of the mouse, and the attribute data of each roller movement in the at least one roller movement.

[0167] It can be seen that the embodiments of the present disclosure can determine the dragging event of the mouse by identifying the movement information of at least one movement of the mouse on the basis of determining the timing of the interactive object pressing the various interactive areas of the mouse; and can identify at least one scroll wheel action of the mouse by determining the attribute data of each scroll wheel action in at least one scroll wheel action, thereby more comprehensively identifying the data input into the host through the mouse.

[0168] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0169] It should be noted that in the embodiments of the present disclosure, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0170] The present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in any of the above-mentioned image processing methods are implemented. The computer-readable storage medium may be transient or non-transient.

[0171] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes part or all of the steps for implementing any one of the above-mentioned image processing methods.

[0172] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above-mentioned image processing method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0173] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device, storage medium, computer program and computer program product embodiments, please refer to the description of the method embodiment of the present disclosure for understanding.

[0174] It should be noted that Fig.11 is a schematic diagram of a hardware entity of an electronic device in an embodiment of the present disclosure, such as Fig.11 As shown, the electronic device 1100 may include: a memory 1101, a processor 1102, and a computer program stored in the memory 1101 and executable on the processor 1102; wherein,

[0175] The processor 1102 is configured to run the computer program to execute any one of the above-mentioned image processing methods.

[0176] In practical applications, the memory 1101 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a ROM, a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1102.

[0177] The embodiment of the present disclosure also provides an information input system, which is communicatively connected to a host, and includes a binocular camera and at least one interactive device. The information input system also includes any one of the above-mentioned image processing devices or any one of the above-mentioned electronic devices.

[0178] The information input system in the embodiment of the present disclosure is different from the virtual keyboard or VR keyboard of laser projection in the related art, which is described in detail below. In order to realize the virtual keyboard of laser projection, it is necessary to customize the emitting light source grid to project a keyboard image outline on the plane, and then track the finger movements through infrared technology to complete the acquisition of input data; it can be seen that the virtual keyboard of laser projection adopts the principle of active projection participation, while the information input system of the embodiment of the present disclosure adopts the passive dual-camera ranging principle, that is, there is a clear difference between the implementation principle of the information input system of the embodiment of the present disclosure and the virtual keyboard of laser projection. The implementation principle of the VR virtual keyboard is to map the real keyboard to the virtual system. It can be seen that in terms of the overall technical solution and the focus of AI processing, the information input system of the embodiment of the present disclosure is different from the VR virtual keyboard.

[0179] Furthermore, the information input system in the embodiment of the present disclosure and the virtual keyboard or VR keyboard of laser projection in the related technology also have the following advantages: when the interactive area is a button, the interactive device has physical buttons, which is conducive to retaining the user's habit of using traditional mechanical keyboard and mouse devices, retaining the operating feel, and improving the user experience to a certain extent; the embodiment of the present disclosure does not require the use of laser projection equipment or VR equipment. Therefore, the information input system of the embodiment of the present disclosure has the advantages of simple implementation, low cost, and low power consumption, which is easy for users to accept.

[0180] The information input system in the disclosed embodiment has at least the following differences compared to a traditional mechanical keyboard and mouse device: 1) The interactive area can be made of elastic materials such as silicone, which is conducive to silent input through the interactive device in the information input system; 2) When the interactive device includes a shell and an interactive area made of elastic material, but does not include springs, switch components, and circuit array structures, the interactive device can be easily disassembled and cleaned, which to a certain extent solves the problem of traditional mechanical keyboard and mouse devices being difficult to clean due to gaps where dirt and grime are hidden.

[0181] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0182] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0183] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0185] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0186] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0187] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0188] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire at least two frames of images captured by a binocular camera, each of the at least two frames of images representing an image of an interactive object and an interactive device, wherein the interactive device comprises a shell and at least two interactive areas made of elastic material and located on the shell; Performing target recognition on the interactive object and the interactive device in the at least two frames of images, and determining a relative position between the interactive object and the at least two interactive areas in each of the at least two frames of images; Determine the timing of the interactive object pressing each of the at least two interactive areas based on the timing of each of the at least two image frames and the relative position of the interactive object and the at least two interactive areas in each of the image frames; Based on the timing of the interaction object pressing each of the at least two interaction areas, input data is determined, where the input data represents data input to a host through the interaction device.

2. The method according to claim 1, characterized in that The method further comprises: Acquire depth information of each pixel in each of the at least two frames of images; The performing target recognition on the interactive object and the interactive device in the at least two frames of images, and determining the relative position of the interactive object and the at least two interactive areas in each of the at least two frames of images, includes: Performing target recognition on the interactive object and the at least two interactive areas in each frame of the image, and determining positions of the interactive object and the at least two interactive areas in each frame of the image; Determine the depth information of the interactive object and the depth information of the at least two interactive areas based on the depth information of each pixel in each frame of the image and the positions of the interactive object and the at least two interactive areas in each frame of the image; Determine the three-dimensional spatial position of the interactive object and the at least two interactive areas based on the positions of the interactive object and the at least two interactive areas in each frame of the image, the depth information of the interactive object and the depth information of the at least two interactive areas; Based on the three-dimensional spatial positions of the interactive object and the at least two interactive areas, a relative position of the interactive object and the at least two interactive areas is determined.

3. The method according to claim 1, characterized in that The determining of input data based on the timing of the interaction object pressing each of the at least two interaction areas comprises: Determining input information corresponding to each pressed interaction area of ​​the at least two interaction areas and a pressing time point of each pressed interaction area based on a timing of the interaction object pressing each interaction area of ​​the at least two interaction areas; Input data is determined based on the input information corresponding to each pressed interaction area and the pressing time point of each pressed interaction area.

4. The method according to any one of claims 1 to 3, characterized in that: The interactive device is a mouse; the method further comprises: Determine movement information of at least one movement of the mouse by performing target tracking on the interactive object and the mouse in the at least two frames of images, wherein the movement information of each movement of the mouse includes position change information, movement start time, and movement end time; The determining of input data based on the timing of the interaction object pressing each of the at least two interaction areas comprises: The input data is determined based on the timing of the interactive object pressing each interactive area of ​​the mouse and the movement information of at least one movement of the mouse.

5. The method according to claim 4, characterized in that The mouse is provided with a scroll wheel; the method further comprises: Determine at least one scroll wheel action achieved by the interactive object by tracking the interactive object and the scroll wheel of the mouse in the at least two frames of images; Determining attribute data of each rolling wheel action in the at least one rolling wheel action, the attribute data comprising: a start time point of the rolling wheel action, a duration of the rolling wheel action, and a rolling wheel speed of the rolling wheel action; Accordingly, the determining the input data based on the timing of the interactive object pressing each interactive area of ​​the mouse and the movement information of at least one movement of the mouse includes: The input data is determined based on the timing of the interactive object pressing each interactive area of ​​the mouse, the movement information of at least one movement of the mouse, and the attribute data of each roller movement in the at least one roller movement.

6. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire at least two frames of images captured by a binocular camera, each of the at least two frames of images representing an image of an interactive object and an interactive device, wherein the interactive device comprises a shell and at least two interactive areas made of elastic material and located on the shell; A first processing module is configured to perform target recognition on the interactive object and the interactive device in the at least two frames of images, determine a relative position between the interactive object and the at least two interactive areas in each frame of the at least two frames of images; and determine a timing of the interactive object pressing each of the at least two interactive areas based on a timing sequence of each frame of the at least two frames of images and a relative position of the interactive object and the at least two interactive areas in each frame of the images. The second processing module is used to determine input data based on the timing of the interactive object pressing each of the at least two interactive areas, where the input data represents data input to the host through the interactive device.

7. An electronic device, characterized in that: The device comprises a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to run the computer program to execute the image processing method according to any one of claims 1 to 5.

8. An information input system, characterized in that: The information input system forms a communication connection with the host, the information input system includes a binocular camera and at least one interactive device, and the information input system also includes the image processing device according to claim 6 or the electronic device according to claim 7.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images