A gesture control method, gesture device and system

Abstract

The embodiment of the present application provides a gesture device, a gesture system and a gesture control method. The gesture device can generate gesture data by sensing hand actions through a sensor, and can send output instructions to a terminal by processing the gesture data through a processor, so as to realize control on the terminal. Thus, the terminal device can be effectively controlled through the gesture device, and the user operation can be more free in space without depending on the operation plane such as a keyboard and a mouse, so that the user experience is better. In addition, the gesture data is directly processed by the gesture device to obtain the output instructions, so that the recognition efficiency can be ensured, the recognition efficiency is improved, the effect of fast response is achieved, and the user experience is further improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a gesture control method, gesture device, and system. Background Art

[0002] In the rapidly changing information age, electronic communications technologies have brought us unprecedented technological experiences, but at the same time, people are increasingly eager for greater freedom in time and space. However, traditional I / O (input / output) devices limit our operating space. For example, computer control often requires a keyboard and mouse, and even the advent of tablets still cannot break free from the limitations of the operating surface. Summary of the Invention

[0003] In view of this, an object of the embodiments of the present application is to provide a gesture control method, a gesture device, and a system, so as to allow users to have more freedom in the operating space.

[0004] An embodiment of the present application provides a gesture control method, including: sensing hand movements to generate gesture data; determining, based on the gesture data and a preset gesture instruction set, an output instruction of a simulated input device corresponding to the gesture data in the gesture instruction set; and sending the output instruction to a terminal.

[0005] In the above implementation process, gesture data is first generated by sensing hand movements, and then the output instructions of the analog input device corresponding to the gesture data in the gesture instruction set are determined based on the gesture data and a preset gesture instruction set, and then the output instructions are sent to the terminal, thereby achieving control of the terminal. In this way, effective control of the terminal device can be achieved only through gesture data corresponding to hand movements, without relying on the limitations of operating planes such as keyboards and mice. This allows users to have greater spatial freedom in operation and a better user experience. In addition, in this application, gesture data is directly processed by the gesture device to obtain output instructions, which can ensure efficient recognition, improve recognition efficiency, achieve a fast response effect, and further enhance the user experience. At the same time, the entire system can be implemented based on standard interfaces and general instructions between hardware systems, while also reducing the amount of data transmitted between devices.

[0006] Furthermore, determining the output instruction of the analog input device corresponding to the gesture data in the gesture instruction set based on the gesture data and a preset gesture instruction set includes: processing the gesture data to parse out a gesture signal that conforms to the gesture instruction set; and determining the output instruction corresponding to the gesture signal in the gesture instruction set based on the current input mode of the analog input device.

[0007] During the implementation described above, gesture data is first screened to determine which of the gesture data is valid (i.e., gesture signals), and then the command is confirmed. This effectively ensures the efficiency of subsequent command confirmation and prevents unnecessary gesture data from interfering with command comparison. Furthermore, during the implementation described above, command confirmation is based on the current input mode of the analog input device. This means that the confirmation of the output command also depends on the current input mode of the analog input device, making the confirmation of the output command more accurate. The selection of the gesture command set takes into account the characteristics of the gesture collector, which primarily measures angles and accelerations, as well as the difficulty in confusing one gesture with another.

[0008] Furthermore, when the simulated input device includes a virtual keyboard, after determining the output instruction of the simulated input device corresponding to the gesture data in the gesture instruction set based on the gesture data and the preset gesture instruction set, it also includes: sending the output instruction related to the virtual keyboard to a keyboard auxiliary device, so that the keyboard auxiliary device can emit light in a specific area according to the received output instruction to prompt the response area or character of the virtual keyboard.

[0009] In the above implementation process, prompts for the user to manipulate the virtual keyboard can be provided, so that the user can better adapt to controlling the terminal through the gesture control method through the keyboard auxiliary device.

[0010] Furthermore, the correspondence between the response area or character of the virtual keyboard and the luminous area on the keyboard auxiliary device includes at least one of the following:

[0011] The left-hand keyboard area of ​​the virtual keyboard responds, and the left-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up;

[0012] The left-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard areas of the corresponding keyboard assistive device light up;

[0013] The right-hand keyboard area of ​​the virtual keyboard responds, and the right-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up;

[0014] The right-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard areas of the corresponding keyboard assistive device light up;

[0015] The uppercase keyboard of the virtual keyboard responds, and the uppercase character lights of the corresponding keyboard auxiliary device are all lit;

[0016] The right-hand numeric keypad area of ​​the virtual keyboard responds, and the right-hand numeric keypad area of ​​the corresponding keyboard auxiliary device lights up;

[0017] The right-hand direction keyboard area of ​​the virtual keyboard responds, and the right-hand direction keyboard area of ​​the corresponding keyboard auxiliary device lights up;

[0018] If a row of the virtual keyboard responds, the row of the corresponding keyboard auxiliary device lights up;

[0019] When a character on the virtual keyboard is responded, the character on the corresponding keyboard auxiliary device is lit;

[0020] The lowercase keyboard of the virtual keyboard responds, and the relevant area of ​​the corresponding keyboard auxiliary device is not lit.

[0021] In the above implementation process, the user can clearly know the current keyboard mode and determine whether the switched keyboard mode is the desired mode.

[0022] Furthermore, the simulated input device includes a virtual keyboard and a virtual mouse, and the gesture instruction set includes a correspondence between gesture data and instructions, and the correspondence includes at least one of the following:

[0023] The corresponding instruction for the gesture data of hand A making a fist and then releasing it is: the switching instruction for hand A's keyboard mode to hand A's mouse mode;

[0024] The corresponding instruction for the gesture data of hand B making a fist and then releasing it is: the instruction for switching hand B's keyboard mode to hand B's mouse mode;

[0025] The gesture data of rotating the left palm by n degrees (n is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switch instruction between an all-lowercase keyboard mode and an all-uppercase keyboard mode, and / or a switch instruction between a right-hand numeric keyboard mode and a directional keyboard mode;

[0026] The gesture data of rotating the right palm by m degrees (m is a preset number greater than 0) and then returning to its original position corresponds to the following instructions: a switch instruction between the right-hand uppercase keyboard mode and the directional keyboard mode, and / or a switch instruction between the right-hand lowercase keyboard mode and the numeric keyboard mode;

[0027] The gesture data of raising or lowering the left or right hand to an angle of β (β is a preset number greater than 0) corresponds to the switching command of switching between keyboard rows;

[0028] The gesture data of hand B's palm rotating m degrees (m is a preset number greater than 0) and then returning to its original position corresponds to the instruction for switching between right-handed flat mouse mode and right-handed vertical mouse mode;

[0029] The gesture data of hand B being raised corresponds to the command for switching between the flat mouse mode and the precise positioning mouse mode for hand B.

[0030] The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise positioning mouse mode to the flat mouse mode;

[0031] The gesture data of hand B moving forward, backward, left, and right corresponds to the following instructions: the movement instructions of hand B's flat mouse moving forward, backward, left, and right;

[0032] The gesture data of hand B being raised corresponds to the instruction for switching hand B's flat mouse mode to precise flat mouse mode.

[0033] The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise flat mouse mode to the flat mouse mode;

[0034] The gesture data of hand B moving up, down, left, and right corresponds to the movement instructions of the vertical mouse of hand B moving up, down, left, and right;

[0035] The gesture data of hand B moving forward corresponds to the instruction for switching hand B's vertical mouse mode to precise vertical mouse mode.

[0036] The gesture data of hand B moving back corresponds to the instruction for switching hand B's precise vertical mouse mode to vertical mouse mode.

[0037] The gesture data of hand A moving up, down, left, right, forward, and backward corresponds to the following commands: the mouse of hand A assists in scrolling up, scrolling down, turning the page forward, turning the page backward, zooming in, and zooming out;

[0038] The gesture data of the back contact between hand A and hand B corresponds to the instruction to power on or off the device.

[0039] In the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; in the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

[0040] The above implementation process is based on ergonomics and defines a gesture instruction set, which is conducive to accurate recognition of gestures and also convenient for users to operate through gestures.

[0041] Furthermore, the gesture control method further includes: determining the control instruction corresponding to the current input mode of the analog input device according to the current input mode of the analog input device and the correspondence between the preset input mode and the control instruction, and sending the control instruction to the terminal.

[0042] The above implementation process automatically determines the corresponding instruction based on the current input mode of the simulated input device, making the mode change of the gesture device more intelligent and providing a better user experience.

[0043] Furthermore, the simulated input device includes a virtual keyboard and a virtual mouse; the correspondence between the input mode and the control instruction includes at least one of the following: when the current input mode is that hand A switches from keyboard mode to mouse mode and hand B is in half-keyboard mode, the control instruction is: a switching instruction for switching hand B's half-keyboard mode to hand B's full-keyboard mode; when the current input mode is that hand A switches from mouse mode to keyboard mode and hand B is in full-keyboard mode, the control instruction is: a switching instruction for switching hand B's full-keyboard mode to hand B's half-keyboard mode; when the current input mode is that hand B switches from keyboard mode to mouse mode and hand A is in half-keyboard mode, the control instruction is: a switching instruction for switching hand A's half-keyboard mode to hand A's full-keyboard mode; when the current input mode is that hand B switches from mouse mode to keyboard mode and hand A is in full-keyboard mode, the control instruction is: a switching instruction for switching hand A's full-keyboard mode to hand A's half-keyboard mode; in the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; in the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

[0044] The above implementation process defines the command correspondence based on the current actual input mode, making the mode change of the gesture device more intelligent and providing a better user experience.

[0045] The present application also provides a gesture device, comprising: a first hand device, the first hand device comprising a processor and a sensor; the sensor is used to sense hand movements and generate gesture data; the processor is used to obtain the gesture data generated by the sensor, determine the output instruction of the analog input device corresponding to the gesture data in the gesture instruction set based on the gesture data and a preset gesture instruction set, and send the output instruction to an external terminal.

[0046] In the above-described implementation structure, the gesture device can sense hand movements through sensors to generate gesture data, which is then processed by a processor to obtain output instructions and sent to the terminal to achieve control of the terminal. In this way, the above-mentioned gesture device can effectively control the terminal device without relying on the limitations of the operating surface such as the keyboard and mouse, allowing users to operate with greater spatial freedom and a better user experience. In addition, in this application, gesture data is directly processed by the gesture device to obtain output instructions, which can ensure recognition accuracy, improve recognition efficiency, achieve a fast response effect, and further enhance the user experience.

[0047] In the first aspect, the gesture device also includes a second hand device and a signal connector; the second hand device includes a processor and a sensor; the processors of the first hand device and the second hand device are respectively communicatively connected to the signal connector, and send the output instruction to the external terminal through the signal connector.

[0048] In the above implementation, the gesture device includes two hand devices, each of which can be placed on the user's hands, which better aligns with the human body. Both hand devices communicate with the terminal via a signal connector, ensuring a better signal quality between the gesture devices and the terminal. Furthermore, the consistent structure of the first and second hand devices facilitates manufacturing processes and reduces production costs.

[0049] Furthermore, the processors of the first hand device and the second hand device both include an acquisition circuit, a motion recognition circuit and a data communicator; the signal connector includes a signal processor; the acquisition circuit collects gesture data from the sensor and sends it to the motion recognition circuit; the motion recognition circuit processes the gesture data, parses out a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor through the data communicator; the signal processor determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device, and sends the output instruction to the external terminal.

[0050] In the above implementation, the output command is determined exclusively by the signal connector, resulting in improved performance and compatibility with current industry interface and signal standards for keyboards and mice. External terminals require no additional installation or configuration, achieving plug-and-play functionality. The first and second hand devices are dedicated to collecting and filtering gesture data, resulting in improved performance and enhanced performance.

[0051] In the second aspect, the gesture device also includes a second hand device; the second hand device includes a processor, a sensor and a first data communicator; the processor of the first hand device is communicatively connected to the processor of the second hand device, and the processor of the first hand device communicates with the external terminal through the first data communicator.

[0052] In the above implementation structure, the gesture device includes two hand devices, which can be placed on the user's two hands respectively, which is more in line with the actual human body. The two hand devices only communicate with the terminal through one of the two hand devices, which can facilitate the terminal to perform recognition and management.

[0053] Furthermore, the processors of the first hand device and the second hand device both include an acquisition circuit, a motion recognition circuit and a second data communicator; the second hand device also includes a signal processor; the acquisition circuit collects gesture data from the sensor and sends it to the motion recognition circuit; the motion recognition circuit processes the gesture data, parses out a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor through the second data communicator; the signal processor determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device, and sends the output instruction to the external terminal through the first data communicator.

[0054] In this implementation, only one signal processor is required in each of the two hand devices to process the instructions corresponding to the gesture signals, reducing structural costs. Furthermore, transmitting gesture recognition data to the signal connector reduces the amount of data transmitted between devices, alleviating the data processing pressure on the signal connector and improving the overall device response speed.

[0055] Furthermore, the sensor includes a finger sensor; the finger sensor can be arranged at a first knuckle and / or a second knuckle of a finger.

[0056] In the above implementation structure, the finger sensor is set at the first knuckle and / or the second knuckle of the finger. In the absence of a fixed plane and no fixed support for the palm and elbow, and limited to the keyboard and mouse simulation scenario, the human finger mainly moves through the first knuckle, and the second knuckle and fingertip are actually passive movements. Therefore, according to the solution of this case, it only needs to be set on the first knuckle, which can reduce the complexity of gestures, improve recognition accuracy and enhance data collection efficiency. In addition, the sensor set at the first knuckle is less intrusive to the user and does not affect the user's other non-keyboard and mouse operations while wearing the device; the hardware structure can shorten the device connection line, which is conducive to the miniaturization of the device and convenient wearing.

[0057] Furthermore, the sensor also includes a palm sensor; the palm sensor is arranged in the processor to sense palm movements to generate gesture data.

[0058] In the above implementation, a specialized palm sensor is used to capture palm motion information (such as forward, backward, left, right, up, down, and flipping), providing the specific gestures required for this application. These gesture designs are based on ergonomics and are designed around the rotational characteristics of human joints. This improves the accuracy of gesture data collection and gesture recognition while also enhancing comfort. Furthermore, this gesture design matches the angle-based calculation model of the posture sensor, enhancing the adaptability of human-machine integration. This is complemented by a spherical matrix keyboard layout design that perfectly matches the human body's elbow-centered movement, making it more user-friendly than conventional planar matrix keyboard designs.

[0059] Furthermore, the gesture device also includes a switch module; the switch module includes: a first power switch, a first Hall sensor and a first magnetic block arranged in the first hand device, and a second power switch, a second Hall sensor and a second magnetic block arranged in the second hand device; when the first hand device and the second hand device approach and separate, the magnetic fields of the first magnetic block and the second magnetic block change, causing the first Hall sensor and the second Hall sensor to generate current and drive the first power switch and the second power switch to turn on or off.

[0060] In the above implementation structure, the user can control the gesture device to turn on or off by bringing the two hand devices close together and separating them. That is, the user can use gestures to turn the power of the gesture device on and off. The user operation is simple, easy to use, and the user experience is high.

[0061] Furthermore, the analog input device includes a virtual keyboard; the virtual keyboard is a spatial spherical matrix structure.

[0062] In the above implementation structure, the virtual keyboard is a spatial spherical matrix structure. When operating, the user can use the elbows and / or wrists of both hands as the origin of the spherical keyboard. The palms can be raised and lowered up and down at a certain angle with the elbows and / or wrists as the origin along with the forearms to locate different keyboard rows. The palms can be swung left and right at a certain angle with the elbows and / or wrists as the origin along with the forearms to locate the keyboard column controlled by each finger. The control is more flexible and can adapt to the characteristics of human body movement to the greatest extent.

[0063] Furthermore, the simulated input device includes a virtual keyboard and a virtual mouse, and the gesture instruction set includes a correspondence between gesture data and instructions, and the correspondence includes at least one of the following:

[0064] The corresponding instruction for the gesture data of hand A making a fist and then releasing it is: the switching instruction for hand A's keyboard mode to hand A's mouse mode;

[0065] The corresponding instruction for the gesture data of hand B making a fist and then releasing it is: the instruction for switching hand B's keyboard mode to hand B's mouse mode;

[0066] The gesture data of rotating the left palm by n degrees (n is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switching instruction between an all-lowercase keyboard mode and an all-uppercase keyboard mode, and / or a switching instruction between a numeric keyboard mode and a directional keyboard mode;

[0067] The gesture data of rotating the right palm by m1 degrees (m1 is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switch instruction between the right-hand uppercase keyboard mode and the directional keyboard mode, and / or a switch instruction between the right-hand lowercase keyboard mode and the numeric keyboard mode;

[0068] The gesture data of raising or lowering the left or right hand to an angle of β (β is a preset number greater than 0) corresponds to the switching command of switching between keyboard rows;

[0069] The gesture data of hand B's palm rotating m2 degrees (m2 is a preset number greater than 0) and then returning to its original position corresponds to the instruction for switching between the right-handed plane mouse mode and the right-handed vertical mouse mode;

[0070] The gesture data of hand B being raised corresponds to the command for switching between the flat mouse mode and the precise positioning mouse mode for hand B.

[0071] The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise positioning mouse mode to the flat mouse mode;

[0072] The gesture data of hand B moving forward, backward, left, and right corresponds to the following instructions: the movement instructions of hand B's flat mouse moving forward, backward, left, and right;

[0073] The gesture data of hand B being raised corresponds to the instruction for switching hand B's flat mouse mode to precise flat mouse mode.

[0074] The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise flat mouse mode to the flat mouse mode;

[0075] The gesture data of hand B moving up, down, left, and right corresponds to the movement instructions of the vertical mouse of hand B moving up, down, left, and right;

[0076] The gesture data of hand B moving forward corresponds to the instruction for switching hand B's vertical mouse mode to precise vertical mouse mode.

[0077] The gesture data of hand B moving back corresponds to the instruction for switching hand B's precise vertical mouse mode to vertical mouse mode.

[0078] The gesture data of hand A moving up, down, left, right, forward, and backward corresponds to the following commands: the mouse of hand A assists in scrolling up, scrolling down, turning the page forward, turning the page backward, zooming in, and zooming out;

[0079] The gesture data of the back contact between hand A and hand B corresponds to the instruction to power on or off the device.

[0080] In the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; in the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

[0081] The above implementation structure is based on ergonomics and defines a gesture instruction set to facilitate user operation through gestures.

[0082] The present application also provides a gesture system, comprising a keyboard auxiliary device and a gesture device of any of the above-mentioned structures; the keyboard auxiliary device is communicatively connected to the gesture device and receives output instructions related to a virtual keyboard sent by the gesture device; the keyboard auxiliary device is provided with a processor and a light-emitting unit; when the processor receives the output instructions related to the virtual keyboard sent by the gesture device, the processor controls the light-emitting unit to emit light in a specific area according to the output instructions to display the response area or character of the virtual keyboard.

[0083] In the above implementation structure, users can effectively control the terminal device through gesture devices, without having to rely on the limitations of operating surfaces such as keyboards and mice. This allows users to operate with greater freedom in space and more ergonomic movements, resulting in a better user experience. In addition, gesture data is directly processed by the gesture device to obtain output instructions, which can ensure recognition accuracy and improve recognition efficiency. The two-level processing of gesture recognition and signal recognition can reduce the amount of data transmission between modules while achieving a fast response effect, further improving the user experience. In addition, the keyboard auxiliary device can be used to prompt the user to operate the virtual keyboard, so that the user can use the keyboard auxiliary device to better adapt to the control of the terminal through the above-mentioned gesture control method.

[0084] Furthermore, the keyboard auxiliary device is a physical style of a preset virtual keyboard; the correspondence between the response area or character of the virtual keyboard and the light-emitting unit on the keyboard auxiliary device includes at least one of the following:

[0085] The left-hand keyboard area of ​​the virtual keyboard responds, and the left-hand keyboard area light-emitting unit of the corresponding keyboard auxiliary device lights up;

[0086] The left-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard area light-emitting units of the corresponding keyboard auxiliary device light up;

[0087] The right-hand keyboard area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up;

[0088] The right-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard area light-emitting units of the corresponding keyboard auxiliary device light up;

[0089] The uppercase keyboard of the virtual keyboard responds, and the uppercase character lights of the corresponding keyboard auxiliary device are all lit;

[0090] The right-hand numeric keypad area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand numeric keypad area of ​​the corresponding keyboard auxiliary device lights up;

[0091] The right-hand direction keyboard area of ​​the virtual keyboard responds, and the right-hand direction keyboard area light-emitting unit of the corresponding keyboard auxiliary device lights up;

[0092] When a row of the virtual keyboard responds, the corresponding light-emitting unit of the keyboard auxiliary device lights up;

[0093] When a character on the virtual keyboard is responded, the corresponding light-emitting unit of the character on the keyboard auxiliary device lights up;

[0094] The lowercase keyboard of the virtual keyboard responds, and the corresponding light-emitting unit in the relevant area of ​​the keyboard auxiliary device is off.

[0095] In the above implementation, the keyboard auxiliary device is a pre-set physical model of a virtual keyboard, allowing the user to easily map the current control position on the virtual keyboard according to the prompts of the keyboard auxiliary device. In addition, the above mapping relationship enables the user to clearly understand the current keyboard mode and determine whether the switched keyboard mode is the desired mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 A basic structural block diagram of a gesture device provided in an embodiment of the present application;

[0098] Figure 2 A schematic diagram of the structure of a gesture device provided in an embodiment of the present application;

[0099] Figure 3A schematic diagram of the structure of another gesture device provided in an embodiment of the present application;

[0100] Figure 4 A structural block diagram of a switch module provided in an embodiment of the present application;

[0101] Figure 5 A structural block diagram of a gesture system provided in an embodiment of the present application;

[0102] Figure 6 A schematic diagram of the structure of a keyboard auxiliary device provided in an embodiment of the present application;

[0103] Figure 7 A basic flow chart of a gesture control method provided in this application;

[0104] Figure 8 This is a schematic diagram of external connections in Example 1 provided in this application;

[0105] Figure 9 A circuit logic diagram of Example 1 provided in this application;

[0106] Figure 10 This is a schematic diagram of external connections in Example 2 provided in this application;

[0107] Figure 11 This is a circuit logic diagram of Example 2 provided in this application;

[0108] Figure 12 A spatial correspondence diagram between keyboard and hand provided in this application;

[0109] Figure 13 A spatial correspondence diagram of a virtual keyboard with a spatial spherical matrix structure provided by this application;

[0110] Figure 14-1 An optional implementation structure of a handheld device provided in this application;

[0111] Figure 14-2 This is an optional implementation structure of a sensor provided in this application. DETAILED DESCRIPTION

[0112] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0113] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0114] Example 1:

[0115] Please see Figure 1 , Figure 1 This is a block diagram of a gesture device provided in an embodiment of the present application. The gesture device 10 includes a first hand device 100, which includes a processor 101 and a sensor 102. Sensor 102 is configured to sense hand movements and generate gesture data. Processor 101 is configured to obtain the gesture data generated by sensor 102, determine the output instruction of the simulated input device corresponding to the gesture data in the gesture instruction set based on the gesture data and a preset gesture instruction set, and transmit the output instruction to an external terminal.

[0116] It should be understood that in actual applications, users are often more accustomed to operating with both hands. Therefore, in this embodiment, the gesture device may also include a second hand device, and the two hand devices are used to operate respectively corresponding to the user's two hands, which is more in line with the human body structure and allows users to have a better operating experience.

[0117] In one possible implementation, see Figure 2 As shown, the gesture device further includes a second hand device 110 and a signal connector 120. The second hand device 110 includes a processor 111 and a sensor 112. The processor 101 of the first hand device 100 and the processor 111 of the second hand device 110 are each communicatively connected to the signal connector 120, and output commands are sent to the external terminal via the signal connector 120. Thus, both hand devices communicate with the terminal via the signal connector 120, ensuring better communication signals between the gesture device and the terminal.

[0118] Exemplarily, the processors of the first handheld device 100 and the second handheld device 110 each include an acquisition circuit (the acquisition circuit in processor 101 is labeled 1011, and the acquisition circuit in processor 111 is labeled 1111), a motion recognition circuit (the motion recognition circuit in processor 101 is labeled 1012, and the motion recognition circuit in processor 111 is labeled 1112), and a data communicator (the data communicator in processor 101 is labeled 1013, and the data communicator in processor 111 is labeled 1113). The signal connector 120 includes a signal processor 121.

[0119] At this time, the acquisition circuit 1011 collects gesture data from the sensor 102 and sends it to the motion recognition circuit 1012. The motion recognition circuit 1012 then processes the gesture data, parses it out, and generates a gesture signal that conforms to the gesture instruction set. It then sends the gesture signal to the signal processor 121 via the data communicator 1013. Similarly, the acquisition circuit 1111 collects gesture data from the sensor 112 and sends it to the motion recognition circuit 1112. The motion recognition circuit 1112 then processes the gesture data, parses it out, and generates a gesture signal that conforms to the gesture instruction set. It then sends the gesture signal to the signal processor 121 via the data communicator 1113. Furthermore, the signal processor 121 determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device and sends the output instruction to the external terminal.

[0120] It should be understood that in actual use, users often perform numerous hand movements, only some of which are actually used to control the terminal. Consequently, among the gesture data generated by the sensor, some are valid gesture data used to generate corresponding output commands to the external terminal, while others are invalid gesture data generated by the user during use and cannot generate corresponding output commands.

[0121] In this approach, the first hand device 100 and / or the second hand device 110 should store a valid gesture data set (gestures in the valid gesture data set are gesture signals that comply with the gesture instruction set) so that when the motion recognition circuit 1012 and / or 1112 processes the gesture data, it can accurately determine which gesture data is valid gesture data (i.e., gesture signals) and which is invalid gesture data and should be ignored. It should be understood that if only one of the first hand device 100 and the second hand device 110 stores a valid gesture data set, the other device needs to obtain the valid gesture data set from the device storing the valid gesture data set through the data communicator when processing the gesture data.

[0122] It should be noted that since the signal processor 121 determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device, to ensure that the signal processor 121 can accurately determine the output instruction corresponding to the gesture signal, a gesture instruction set should be pre-stored in the signal connector 120, and the gesture instruction set should include the correspondence between gesture data and instructions. It should also be noted that to ensure that the output instruction determined by the signal processor 121 can be smoothly transmitted to an external terminal, the signal connector 120 should also be equipped with a data communicator, which is connected to the signal processor 121 to enable the reception of gesture signals from the first hand device 100 and the second hand device 110 and the transmission of output instructions.

[0123] It should be noted that in the above example, the internal structures of the first hand device 100 and the second hand device 110 can be consistent, so that the first hand device 100 and the second hand device 110 can be manufactured better in the process flow, making the production cost of the gesture device lower.

[0124] Of course, in the specific implementation process, it may not be completely consistent. For example, in another possible implementation method of the embodiment of the present application, see Figure 3 As shown, the gesture device also includes a second hand device 110; wherein the second hand device 110 includes a processor 111, a sensor 112 and a first data communicator 1113; the processor 101 of the first hand device 100 and the processor 111 of the second hand device 110 are communicatively connected, and the processor 101 of the first hand device 100 and the processor 111 of the second hand device 110 both communicate with the external terminal through the first data communicator 1113 of the second hand device 110 to send the output instruction to the external terminal. In this way, both hand devices communicate with the terminal through the first data communicator 1113 in the second hand device 110. For the terminal, the instructions it receives are all from the second hand device 110. When managing, it only needs to record the identification code of the second hand device 110 to effectively manage the entire gesture device, thereby simplifying the management complexity and improving the management efficiency.

[0125] Optionally, the processors of the first handheld device 100 and the second handheld device 110 both include an acquisition circuit (the acquisition circuit in processor 101 is labeled 1011, and the acquisition circuit in processor 111 is labeled 1111), a motion recognition circuit (the motion recognition circuit in processor 101 is labeled 1012, and the motion recognition circuit in processor 111 is labeled 1112), and a second data communicator (the second data communicator in processor 101 is labeled 1013, and the second data communicator in processor 111 is labeled 1113). In addition, the second handheld device 110 also includes a signal processor 113.

[0126] At this time, the acquisition circuit 1011 collects gesture data from the sensor 102 and sends it to the motion recognition circuit 1012. The motion recognition circuit 1012 then processes the gesture data, parses it into a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor 113 of the second hand device 110 via the second data communicator 1013. Similarly, the acquisition circuit 1111 collects gesture data from the sensor 112 and sends it to the motion recognition circuit 1112. The motion recognition circuit 1112 then processes the gesture data, parses it into a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor 113 via the second data communicator 1113. Furthermore, the signal processor 113 determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device, and sends the output instruction to the external terminal via the first data communicator 1113.

[0127] In this example, a valid gesture data set should be stored in the first hand device 100 and / or the second hand device 110 (the gestures in the valid gesture data set are gesture signals that comply with the gesture instruction set) so that when the motion recognition circuit 1012 and / or 1112 processes the gesture data, it can accurately know which gesture data are valid gesture data (i.e., gesture signals) and which are invalid gesture data that should be ignored.

[0128] It should be noted that since the signal processor 113 will determine the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device, in order to ensure that the signal processor 113 can accurately determine what the output instruction corresponding to the gesture signal is, in addition to storing a valid gesture data set, the second hand device 110 should also store a gesture instruction set, and the gesture instruction set should include the correspondence between gesture data and instructions.

[0129] It should be understood that in the above two examples, the acquisition circuit and the motion recognition circuit can be implemented by a single-chip microcomputer such as Nuc442 or STM32. The sensor can be implemented by a gesture acquisition chipset such as MPU9250, MINI54FDE, etc. The data communicator can be a communication module such as NFC (Near Field Communication), mrf24l01, Bluetooth, WiFi, etc. For example, see Figure 14-1 and 14-2 The circuit shown is implemented as follows, Figure 14-1 It is an optional implementation structure of the hand device, and Figure 14-2It is an optional implementation structure of the sensor. It should be noted that in the first example above, the signal processor 121 in the signal connector 120 can also be implemented by a single-chip microcomputer Nuc442 or STM32. In this case, the signal connector 120 can be obtained by simply connecting a Bluetooth communicator, a data storage device, etc. through the single-chip microcomputer Nuc442 or STM32.

[0130] In the embodiment of the present application, the sensor 102 / 112 may include a finger sensor 1021 / 1121, and the finger sensor 1021 / 1121 may be set at the first knuckle and / or the second knuckle of the finger. It should be noted that in the embodiment of the present application, the number of finger sensors 1021 / 1121 in the first hand device 100 or the second hand device 110 may be 5, so that when the user is using it, a finger sensor is placed on each finger to specifically sense the movement of that finger. In this way, the movement of the five fingers on one hand of the user is sensed by five finger sensors, and the generated gesture data is more accurate. In some use cases where accuracy is not required, the number of finger sensors 1021 / 1121 in the first hand device 100 or the second hand device 110 in the embodiment of the present application may be 2, 3 or 4.

[0131] In this embodiment, the sensors 102 / 112 may also include a palm sensor 1022 / 1122. In this embodiment, the palm sensor 1022 / 1122 may be incorporated into the processor. Thus, when a user uses the gesture device, the palm sensor 1022 / 1122 is positioned against the back of the user's hand or palm, sensing the user's palm movements and generating corresponding gesture data. This allows the palm sensor 1022 / 1122 to clearly and accurately detect the user's overall hand movements, such as forward, backward, left, right, lift, lower, and flip, without being affected by changes in finger movements. This ensures accurate perception of overall hand movements. Of course, it should be understood that when the palm sensor 1021 / 1121 is not set but only the finger sensor 1021 / 1121 is set, gesture data corresponding to the overall hand movement can also be obtained through the overall analysis of multiple finger sensors 1021 / 1121, but this method is more complicated for data processing and screening, and the accuracy is not as high as setting the palm sensor 1021 / 1121.

[0132] It should be noted that the finger sensor and / or palm sensor in the embodiment of the present application can adopt a nine-axis gesture sensor, and then use the nine-axis gesture sensor to effectively sense hand movements. Of course, it can also be implemented using sensor devices such as a six-axis gesture sensor.

[0133] It should be noted that, in the embodiment of the present application, when the gesture device has two hand devices, in order to facilitate the user to realize the opening and closing control of the two hand devices, a switch module can be set in the gesture device, see Figure 4 As shown, the switch module includes: a first power switch 106, a first Hall sensor 105, and a first magnetic block 104 provided in the first handheld device 100; and a second power switch 116, a second Hall sensor 115, and a second magnetic block 114 provided in the second handheld device 110. When the first handheld device 100 and the second handheld device 110 approach and then separate, the magnetic fields of the first magnetic block 104 and the second magnetic block 114 change, causing the first Hall sensor 105 and the second Hall sensor 115 to generate current, driving the first power switch 106 and the second power switch 116 to turn on or off. This allows the user to turn the power on and off with gestures, which is simple to operate, convenient to use, and provides a high user experience.

[0134] In an embodiment of the present application, the simulated input device may include a virtual mouse and a virtual keyboard. The gesture instruction set includes a correspondence between gesture data and instructions, and the correspondence includes at least one of the following:

[0135] The gesture data of hand A making a fist and then releasing it corresponds to the instruction for switching from hand A's keyboard mode to hand A's mouse mode.

[0136] It should be noted that the left-handed keyboard mode mentioned above refers to the mode in which the left hand can control the keyboard. Similarly, the right-handed keyboard mode mentioned below refers to the mode in which the right hand can control the keyboard. The left-handed mouse mode refers to the mode in which the left hand can control the mouse. Similarly, the right-handed mouse mode refers to the mode in which the right hand can control the mouse.

[0137] The gesture data of hand B making a fist and then releasing it corresponds to the instruction for switching from hand B's keyboard mode to hand B's mouse mode.

[0138] The gesture data of rotating the left palm n degrees (n is a preset number greater than 0) and then returning to its original position corresponds to the following instructions: a switching instruction between the all-lowercase keyboard mode and the all-uppercase keyboard mode, and / or a switching instruction between the numeric keyboard mode and the directional keyboard mode.

[0139] The gesture data of rotating the right palm m1 (m1 is a preset number greater than 0) degrees and then returning to its original position corresponds to: a switching instruction between the right-hand uppercase keyboard mode and the directional keyboard mode, and / or a switching instruction between the right-hand lowercase keyboard mode and the numeric keyboard mode.

[0140] It should be noted that, in this embodiment, the numeric keyboard can be controlled by the right hand.

[0141] The gesture data of hand B's palm rotating m2 (m2 is a preset number greater than 0) degrees and then returning to its original position corresponds to the instruction for switching between the right-hand plane mouse mode and the right-hand vertical mouse mode.

[0142] It should be noted that there is no clear size relationship between n, m1 and m2 in the present application. In a feasible implementation of this embodiment, n, m1 and m2 can be equal.

[0143] The gesture data of the right hand being raised corresponds to the instruction for switching between the flat mouse mode and the precise positioning mouse mode of hand B.

[0144] The corresponding instruction for the gesture data of hand B putting down is: the switching instruction for hand B to switch from precise positioning mouse mode to flat mouse mode.

[0145] The gesture data of raising or lowering the left or right hand by an angle of β (β is a preset number greater than 0) corresponds to a switching instruction for switching between keyboard rows.

[0146] It is particularly important to note that in actual applications, to achieve precise control of the virtual keyboard by the gesture device, it is necessary to accurately locate which keyboard key corresponds to the current gesture action of the device. To this end, the following methods can be used in the embodiments of the present application to determine which key on the virtual keyboard corresponds to the finger when the user performs a gesture operation:

[0147] Method 1: Filter and integrate the data measured by the sensor's accelerometer to determine the distance the finger has moved, thereby obtaining the finger's coordinates. The corresponding key can then be determined by mapping the finger coordinates to the coordinates of the virtual keyboard. However, this method still uses the fixed keyboard concept. The finger position obtained through integration is necessarily based on the traditional key positions, and the user's movement distance cannot exceed the range defined by the keyboard.

[0148] In this regard, the embodiment of the present application also provides a second method. In the second method, we predefine which columns of the keyboard can be controlled by each finger, and control the switching of keyboard rows by gesture data of hand A or hand B raising or lowering an angle of β (β is a preset number greater than 0). Optionally, the angle range of the palm raised and lowered on the vertical plane with the wrist or elbow as the origin can be used to determine which keyboard row is enabled by the left and right hands. When the currently enabled keyboard row is determined, the keyboard key position corresponding to each finger is also determined. In particular, when it comes to a finger corresponding to two or more columns of keys, the keyboard column enabled by the left and right hands can be determined based on the angle of the palm rotated on the horizontal plane with the wrist or elbow as the origin. For example, see Figure 12As shown, when entering the keyboard mode, the angle α between the left / right hand palm on the vertical plane and the horizontal plane is the initial angle, corresponding to the Y0 row. Thereafter, when the left / right hand palm is deflected by -β, β to 3β angles relative to α on the vertical plane, it corresponds to the Y-1 row and Y1 to Y3 rows respectively. Similarly, the switching of keyboard columns can also be achieved through angles. It is particularly important to note that in this embodiment, when switching back to the keyboard mode from the mouse mode, the current position of the palm can be used as the initial position to locate the initial row of the virtual keyboard, for example, to the Y0 row. Thereafter, each deflection of β angle relative to the initial position corresponds to switching a row.

[0149] It should be understood that the above two methods of determining the keyboard key positions corresponding to gesture data are only two feasible implementation methods provided in this embodiment, and do not mean that the embodiments of this application can only be implemented using the above two methods. In fact, as long as the method can accurately determine the keyboard key positions corresponding to gesture data, it can be adopted by the embodiments of this application and should be within the scope of protection of this application.

[0150] The gesture data of hand B moving forward, backward, left, and right corresponds to the movement instructions of hand B's flat mouse moving forward, backward, left, and right.

[0151] The gesture data of hand B being raised corresponds to the instruction for switching the flat mouse mode of hand B to the precise flat mouse mode.

[0152] The corresponding instruction for the gesture data of hand B putting down is: the switching instruction for hand B to switch from the precise flat mouse mode to the flat mouse mode.

[0153] The gesture data of hand B moving forward corresponds to the instruction for switching hand B's facade mouse mode to precise facade mouse mode.

[0154] The gesture data of hand B moving back corresponds to the instruction for switching hand B's precise elevation mouse mode to elevation mouse mode.

[0155] The gesture data of hand B moving up, down, left, and right corresponds to the movement instructions of the vertical mouse of hand B moving up, down, left, and right.

[0156] The gesture data of hand B moving up, down, left, right, forward and backward corresponds to the following instructions: the mouse of hand A assists in scrolling up, scrolling down, turning the page forward, turning the page backward, zooming in and out.

[0157] The gesture data of the back contact between hand A and hand B corresponds to the instruction for turning on or off the device.

[0158] In the above keyboard mode, the virtual keyboard is divided into the left-hand keyboard area and the right-hand keyboard area, and the left-hand keyboard area and the right-hand keyboard area cannot be swapped; in mouse mode, hand A can be the left hand and hand B can be the right hand, or hand A can be the right hand and hand B can be the left hand.

[0159] It should be understood that in the embodiment of the present application, the signal processor determines the output instruction corresponding to the received gesture signal in the gesture instruction set based on the current state of the analog input device. For example, when hand A is the left hand and hand B is the right hand, and the gesture instruction set includes the correspondence between all the above gesture data and instructions: when the gesture signal is the left palm rotating n degrees and then returning to its original position, if the current corresponding keyboard is a lowercase keyboard, the corresponding instruction is: switch the lowercase keyboard to the uppercase keyboard; similarly, if the current corresponding keyboard is an uppercase keyboard, the corresponding instruction is: switch the uppercase keyboard to the lowercase keyboard; and if the current keyboard has a numeric keypad, the corresponding instruction is: switch the numeric keypad to the directional keyboard; similarly, if the current keyboard has a directional keyboard, the corresponding instruction is: switch the directional keyboard to the numeric keypad. When the gesture signal is the right palm rotating m degrees and then returning to its original position, if the current corresponding keyboard is a right-hand lowercase keyboard, the corresponding instruction is: switch the right-hand lowercase keyboard to the numeric keyboard. Similarly, when the current corresponding keyboard is a numeric keyboard, the corresponding instruction is: switch the numeric keyboard to the right-hand lowercase keyboard; if the current corresponding keyboard is a right-hand uppercase keyboard, the corresponding instruction is: switch the right-hand uppercase keyboard to the directional keyboard. Similarly, when the current corresponding keyboard is a directional keyboard, the corresponding instruction is: switch the directional keyboard to the right-hand uppercase keyboard.

[0160] It should be understood that, in the embodiment of the present application, for the gesture data of rotating the palm of hand A n degrees and then returning to its original position or rotating the palm of hand B m degrees and then returning to its original position, the values ​​of n and m will affect the sensitivity of the user's control. If the values ​​of n and m are too small (such as 1-10), the control will be too sensitive, and the user's palm may just shake slightly to meet the requirements of rotating the palm of hand A n degrees and then returning to its original position or rotating the palm of hand B m degrees and then returning to its original position, which can easily cause the user to make an erroneous operation and reduce the user experience. When the values ​​of n and m are too large (such as 180), it will be inconvenient for the user to realize the operation of rotating the palm of hand A n degrees and then returning to its original position or rotating the palm of hand B m degrees and then returning to its original position, thereby affecting the user's experience. Optionally, in a feasible implementation of the embodiment of the present application, the values ​​of n and m can be taken as 90, so that the user operation is easier and less prone to erroneous operation.

[0161] It should be noted that in the embodiments of the present application, the control instruction corresponding to the current input mode of the analog input device can be determined based on the current input mode of the analog input device and the preset correspondence between the input mode and the control instruction, and the control instruction can be sent to the terminal. This makes the mode change of the gesture device more intelligent and the user experience better.

[0162] Exemplarily, the correspondence between the input mode and the control instruction may include at least one of the following:

[0163] When the current input mode is that hand A switches from keyboard mode to mouse mode and hand B is in half-keyboard mode, the control instruction is: switching instruction from hand B half-keyboard mode to hand B full-keyboard mode;

[0164] It should be noted that in the embodiments of the present application, the virtual keyboard can be divided into two areas, left and right, corresponding to the user's left and right hands for operation respectively. The A-hand half-keyboard mode refers to the mode in which hand A can control the keyboard in one area of ​​the virtual keyboard, and the B-hand half-keyboard mode refers to the mode in which hand B can control the keyboard in another area of ​​the virtual keyboard. The A-hand full-keyboard mode refers to the mode in which hand A can control the keyboard in all areas of the virtual keyboard, and the B-hand full-keyboard mode refers to the mode in which hand B can control the keyboard in all areas of the virtual keyboard.

[0165] When the current input mode is that hand A switches from mouse mode to keyboard mode and hand B is in full keyboard mode, the control instruction is: switching instruction from hand B full keyboard mode to hand B half keyboard mode;

[0166] When the current input mode is that hand B switches from keyboard mode to mouse mode and hand A is in half-keyboard mode, the control instruction is: switching instruction from hand A's half-keyboard mode to hand A's full-keyboard mode;

[0167] When the current input mode is that hand B switches from mouse mode to keyboard mode and hand A is in full keyboard mode, the control instruction is: switching instruction from full keyboard mode of hand A to half keyboard mode of hand A;

[0168] In the above keyboard mode, the virtual keyboard is divided into the left-hand keyboard area and the right-hand keyboard area, and the left-hand keyboard area and the right-hand keyboard area cannot be swapped; in mouse mode, hand A can be the left hand and hand B can be the right hand, or hand A can be the right hand and hand B can be the left hand.

[0169] In this way, when one hand of the user no longer controls the keyboard, the control authority of the other hand automatically becomes able to control the entire keyboard, which ensures that the user has a good control experience.

[0170] In the embodiment of the present application, the virtual keyboard can be designed as a common planar matrix structure, but can also be arranged as a spatial spherical matrix structure, for example Figure 13 The structure shown. When the virtual keyboard is a spatial spherical matrix structure, the positioning of the virtual keyboard output instructions can be well achieved. For example, the elbows and / or wrists of both hands can be used as the origin of the spherical keyboard. When the palms follow the forearms to lift and lower the elbows and / or wrists to a certain angle with the elbows and / or wrists as the origin, different keyboard rows can be positioned. When the palms follow the forearms to swing left and right with the elbows and / or wrists as the origin, the keyboard columns controlled by each finger can be positioned. This makes the user's control more flexible and can adapt to the characteristics of human body movements to the greatest extent. It should be noted that Figure 13 The keys in the dotted box are the keys controlled by the left and right hands respectively in half-keyboard mode. The keys outside the dotted box are the keys of the other half of the keyboard extended by the left or right hand in full keyboard mode.

[0171] It should be noted that, in this embodiment, when switching from mouse mode back to keyboard mode, the current position of the palm (for example, represented by a gesture angle) can be used as the initial position to locate the initial row of the virtual keyboard.

[0172] It should be understood that the various hand devices provided in the embodiments of the present application can be provided in gloves of various structural forms, or can be provided in the form of patches, etc.

[0173] It should be noted that in actual applications, since the user directly uses the gesture device to control the virtual keyboard, and the virtual keyboard may not be displayed, the user is actually performing blind keyboard operation. For those who are not familiar with the device, they may not be used to the operation or make mistakes when performing blind keyboard operation. In this regard, a gesture system is also provided in this embodiment, see Figure 5 As shown, the gesture system includes a gesture device 10 of the above structure and a keyboard auxiliary device 20, and the keyboard auxiliary device 20 is in communication with the gesture device 10 and receives output instructions related to the virtual keyboard sent by the gesture device. The keyboard auxiliary device 20 is provided with a processor 201 and a light-emitting unit 202. When the processor 201 receives the output instructions related to the virtual keyboard sent by the gesture device 10, it can control the light-emitting unit 202 to emit light in a specific area according to the output instructions to display the response area or characters of the virtual keyboard. In this way, the keyboard auxiliary device 20 can prompt the user's manipulation of the virtual keyboard accordingly, so that the user can better adapt to the control of the terminal through the above-mentioned gesture device method through the keyboard auxiliary device.

[0174] For example, in this embodiment, the light-emitting unit 202 can use multiple controlled LED lights, and the processor can control the LED lights in different positions to turn on and off to display the response area or characters of the virtual keyboard. In this embodiment of the application, the processor of the keyboard auxiliary device 20 can be implemented by a single-chip microcomputer or chip such as NUC442 or STM32F407, and it can be connected to a communicator such as Bluetooth and a data storage device to obtain the keyboard auxiliary device 20 of the present application.

[0175] Optionally, in order to ensure that the prompts the user sees on the keyboard auxiliary device 20 accurately correspond to the virtual keyboard, in this embodiment, the style of the keyboard auxiliary device can be designed to be the physical style of a preset virtual keyboard, which includes all the keyboard areas of the virtual keyboard, and the layout of the keyboard areas is the same as that of the virtual keyboard. It is particularly important to note that the numeric keypad area in the virtual keyboard can overlap with the right-hand or left-hand keyboard area in the virtual keyboard, and the keyboard can be switched by switching. However, when corresponding to the physical keyboard, these two keyboard areas need to correspond to two different physical locations. Therefore, the keyboard auxiliary device 20 can be provided with a separate numeric keypad area, for example, see Figure 6 The structural style of the keyboard auxiliary device 20 is shown.

[0176] In this embodiment, the correspondence between the response area or character of the virtual keyboard and the light emitting unit 202 on the keyboard auxiliary device 20 includes at least one of the following:

[0177] The left-hand keyboard area of ​​the virtual keyboard responds, and the corresponding left-hand keyboard area light-emitting unit of the keyboard auxiliary device 20 lights up.

[0178] It should be noted that when switching to the left-hand half keyboard mode, the left-hand keyboard area of ​​the virtual keyboard responds.

[0179] The left-hand full keyboard area of ​​the virtual keyboard responds, and the corresponding light-emitting units of the left-hand and right-hand keyboard areas of the keyboard auxiliary device 20 light up.

[0180] It should be noted that when switching to the left-hand full keyboard mode, the left-hand full keyboard area of ​​the virtual keyboard responds.

[0181] The right-hand keyboard area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand keyboard area of ​​the corresponding keyboard auxiliary device 20 lights up.

[0182] It should be noted that when switching to the right-hand half keyboard mode, the right-hand keyboard area of ​​the virtual keyboard responds.

[0183] The right-hand full keyboard area of ​​the virtual keyboard responds, and the corresponding light-emitting units of the left-hand and right-hand keyboard areas of the keyboard auxiliary device 20 light up.

[0184] It should be noted that when switching to the right-hand full keyboard mode, the right-hand full keyboard area of ​​the virtual keyboard responds.

[0185] The uppercase keyboard of the virtual keyboard responds, and the uppercase character lights of the corresponding keyboard auxiliary device 20 are all lit. It should be noted that when switching to the uppercase keyboard mode, the uppercase keyboard of the virtual keyboard responds.

[0186] The right-hand numeric keypad area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand numeric keypad area of ​​the corresponding keyboard auxiliary device 20 lights up.

[0187] It should be noted that when switching to the numeric keyboard mode, the numeric keyboard area of ​​the virtual keyboard responds.

[0188] The right-hand direction keyboard area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand direction keyboard area of ​​the corresponding keyboard auxiliary device 20 lights up.

[0189] It should be noted that when switching to the directional keyboard mode, the directional keyboard area of ​​the virtual keyboard responds.

[0190] When a row of the virtual keyboard responds, the light-emitting units of the corresponding row of the keyboard auxiliary device 20 light up.

[0191] When a character on the virtual keyboard responds, the light-emitting unit of the corresponding character on the keyboard auxiliary device 20 lights up.

[0192] The lowercase keyboard of the virtual keyboard responds, and the corresponding light-emitting unit in the relevant area of ​​the keyboard auxiliary device 20 is off.

[0193] It should be noted that when switching to lowercase keyboard mode, the lowercase keyboard of the virtual keyboard responds.

[0194] Based on the above implementation structure, the present application embodiment also provides a gesture control method, see Figure 7 As shown, Figure 7 A basic flow chart of a gesture control method provided in an embodiment of the present application includes:

[0195] S701: Sense hand movements and generate gesture data;

[0196] In this embodiment, the sensor in the above implementation structure can be used to sense hand movements and generate gesture data.

[0197] S702: Determine, based on the gesture data and a preset gesture instruction set, an output instruction of the analog input device corresponding to the gesture data in the gesture instruction set;

[0198] In this embodiment, the gesture data may be processed first to parse out a gesture signal that complies with a gesture instruction set, and then the output instruction corresponding to the gesture signal in the gesture instruction set may be determined according to the current input mode of the analog input device.

[0199] It should be noted that in the embodiment of the present application, a valid gesture data set can be pre-set (the gestures in the valid gesture data set are gesture signals that comply with the gesture instruction set) so that gesture signals that comply with the gesture instruction set can be quickly parsed from the generated gesture data.

[0200] It should also be noted that, in the embodiment of the present application, the same gesture signal may correspond to multiple different instructions, and which instruction the current gesture signal specifically corresponds to needs to be determined based on the current input mode of the analog input device (in this embodiment, the analog input device may include a virtual mouse and / or a virtual keyboard). For example, when hand A is the left hand and hand B is the right hand, and the gesture instruction set includes the correspondence between all the above-mentioned gesture data and instructions: when the gesture signal is the left palm rotating n degrees and then returning to its original position, if the current corresponding keyboard is a lowercase keyboard, the corresponding instruction is: switch the lowercase keyboard to the uppercase keyboard; similarly, when the current corresponding keyboard is an uppercase keyboard, the corresponding instruction is: switch the uppercase keyboard to the lowercase keyboard; and when the current keyboard has a numeric keypad, the corresponding instruction is: switch the numeric keypad to the directional keyboard; similarly, when the current keyboard has a directional keyboard, the corresponding instruction is: switch the directional keyboard to the numeric keypad. It should be noted that the specific content of the gesture instruction set has been described in the above content and will not be repeated here.

[0201] S703: Send the output instruction to the terminal.

[0202] In an embodiment of the present application, when the simulated input device includes a virtual keyboard, after determining the output instruction of the simulated input device corresponding to the gesture data in the gesture instruction set based on the gesture data and a preset gesture instruction set, the output instruction related to the virtual keyboard can also be sent to the keyboard auxiliary device, so that the keyboard auxiliary device can illuminate a specific area to indicate the response area or character of the virtual keyboard based on the received output instruction. It should be noted that the correspondence between the response area or character of the virtual keyboard and the illuminated area on the keyboard auxiliary device has been described above and will not be repeated here.

[0203] In an embodiment of the present application, the gesture control method may further include: determining a control instruction corresponding to the current input mode of the analog input device based on the current input mode of the analog input device and a preset correspondence between the input mode and the control instruction, and sending the control instruction to the terminal. This makes the mode change of the gesture device more intelligent and provides a better user experience.

[0204] Exemplarily, the correspondence between the input mode and the control instruction may include at least one of the following:

[0205] When the current input mode is that hand A switches from keyboard mode to mouse mode and hand B is in half-keyboard mode, the control instruction is: switching instruction from hand B half-keyboard mode to hand B full-keyboard mode;

[0206] It should be noted that in the embodiments of the present application, the virtual keyboard can be divided into two areas, left and right, corresponding to the user's left and right hands for operation respectively. The A-hand half-keyboard mode refers to the mode in which hand A can control the keyboard in one area of ​​the virtual keyboard, and the B-hand half-keyboard mode refers to the mode in which hand B can control the keyboard in another area of ​​the virtual keyboard. The A-hand full-keyboard mode refers to the mode in which hand A can control the keyboard in all areas of the virtual keyboard, and the B-hand full-keyboard mode refers to the mode in which hand B can control the keyboard in all areas of the virtual keyboard.

[0207] When the current input mode is that hand A switches from mouse mode to keyboard mode and hand B is in full keyboard mode, the control instruction is: switching instruction from hand B full keyboard mode to hand B half keyboard mode;

[0208] When the current input mode is that hand B switches from keyboard mode to mouse mode and hand A is in half-keyboard mode, the control instruction is: switching instruction from hand A's half-keyboard mode to hand A's full-keyboard mode;

[0209] When the current input mode is that hand B switches from mouse mode to keyboard mode and hand A is in full keyboard mode, the control instruction is: switching instruction from full keyboard mode of hand A to half keyboard mode of hand A;

[0210] In the above keyboard mode, the virtual keyboard is divided into the left-hand keyboard area and the right-hand keyboard area, and the left-hand keyboard area and the right-hand keyboard area cannot be swapped; in mouse mode, hand A can be the left hand and hand B can be the right hand, or hand A can be the right hand and hand B can be the left hand.

[0211] In this way, when one hand of the user no longer controls the keyboard, the control authority of the other hand automatically becomes able to control the entire keyboard, which ensures that the user has a good control experience.

[0212] In summary, the gesture device, gesture system and gesture control method provided by the embodiments of the present application are such that the gesture device can sense hand movements through sensors to generate gesture data, and then process the data through a processor to obtain output instructions and send them to the terminal to achieve control of the terminal. In this way, effective control of the terminal device can be achieved through the above-mentioned gesture device, without relying on the limitations of the operating plane such as the keyboard and mouse, which can enable the user to operate with greater freedom in space and provide a better user experience. In addition, in the present application, the gesture data is directly processed by the gesture device to obtain output instructions, which can ensure efficient recognition, improve recognition efficiency, achieve a fast response effect, and further enhance the user experience. In addition, the gesture system also includes a keyboard auxiliary device, which can provide prompts to the user on how to operate the virtual keyboard, so that the user can be more adaptable to the above-mentioned gesture control method to achieve control of the terminal through the keyboard auxiliary device.

[0213] Example 2:

[0214] This embodiment is based on the first embodiment. Figure 2 and Figure 3 The present application is further illustrated by taking the example structure of as an example.

[0215] Example 1: See Figure 8 As shown, the first hand device 100 and the second hand device 110 are both designed to be installed on the back of the hand, which does not affect the normal use of the palm direction. The first hand device 100 and the second hand device 110 are symmetrically designed, and nine-axis posture sensors are installed at the corresponding positions on the back of the first knuckles of the five fingers. The five sensors are connected to the processor 101 / 111 in the hand device located on the back of the hand through wires. The first hand device 100, the second hand device 110 and the keyboard auxiliary device 20 are connected to the signal connector 120 in a wireless manner, and are connected to the terminal (such as a computer, tablet or mobile phone, etc.) through a data communicator such as Bluetooth. Among them, the keyboard auxiliary device 20 can be used to display the signal status of the virtual keyboard output to prompt the response area or character of the virtual keyboard.

[0216] See also Figure 9 As shown, the first handheld device 100 and the second handheld device 110 each include a power switch driven by a Hall effect sensor and a magnetic block. Because the two Hall effect sensors are fixed in position, no induced current is generated when they are far apart. However, when the first handheld device 100 and the second handheld device 110 move closer and then further apart, the magnetic field changes, causing the Hall effect sensor to generate current, which then turns the power switch on and off.

[0217] After the first hand device 100 and the second hand device 110 are turned on, the sensors L0~L5 and R0~R5 generate the gesture change data of the palm and fingers in real time, and after being collected by the acquisition circuit 1011 / 1111, transmit it to the motion recognition circuit 1012 / 1112. The motion recognition circuit 1012 / 1112 processes and analyzes the gesture data, parses out the gesture signal that conforms to the gesture instruction set, and transmits it to the signal processor 121 through the data communicator 1013 / 1113. It should be noted that a data communicator 122 should be provided in the signal connector 120 for data communication with the first hand device 100 and the second hand device 110 and for communication with the terminal. In addition, a memory 123 should also be provided in the signal connector 120 for storing the gesture instruction set. Accordingly, a valid gesture data set can be stored in the first hand device 100 and the second hand device 110 (the gestures in the valid gesture data set are gesture signals that conform to the gesture instruction set), for example Figure 9 The valid gesture data set can be stored in the memory of the first hand device 100 and the second hand device 110. The signal processor 121 distinguishes the gesture signals and sends the output instructions to the terminal through the data communicator 122 and the terminal connector, and at the same time transmits the instructions related to the virtual keyboard to the keyboard auxiliary device 20. It should be noted that a memory can be set in the signal connector 120 to store the gesture instruction set to ensure that the signal processor 121 processes the gesture signals to obtain the corresponding output instructions. It should also be noted that Figure 9 The data communicator in the signal connector 120 can use only one to realize communication with multiple devices through function multiplexing, or it can be as follows: Figure 9 As shown in FIG, multiple data communicators are used to respectively realize data communication with different devices.

[0218] Example 2: See Figure 10 As shown, the first hand device 100 and the second hand device 110 are both designed to be installed on the back of the hand, which does not affect the normal use of the palm direction. The first hand device 100 and the second hand device 110 are symmetrically designed, and nine-axis posture sensors are installed at the corresponding positions on the back of the first knuckles of the five fingers. The five sensors are connected to the processor 101 / 111 in the hand device located on the back of the hand through wires. The first hand device 100 and the second hand device 110 and the keyboard auxiliary device 20 are connected to the second hand device 110 wirelessly, and are connected to the terminal (such as a computer, tablet or mobile phone, etc.) through a data communicator such as Bluetooth of the second hand device 110. Among them, the keyboard auxiliary device 20 can be used to display the signal status of the virtual keyboard output to prompt the response area or character of the virtual keyboard.

[0219] See also Figure 11As shown, the first handheld device 100 and the second handheld device 110 each include a power switch driven by a Hall effect sensor and a magnetic block. Because the two Hall effect sensors are fixed in position, no induced current is generated when they are far apart. However, when the first handheld device 100 and the second handheld device 110 move closer and then further apart, the magnetic field changes, causing the Hall effect sensor to generate current, which then turns the power switch on and off.

[0220] After the first hand device 100 and the second hand device 110 are turned on, sensors L0-L5 and R0-R5 generate real-time data on changes in palm and finger gestures. This data is collected by acquisition circuits 1011 / 1111 and then transmitted to motion recognition circuits 1012 / 1112. Motion recognition circuits 1012 / 1112 process and analyze the gesture data to parse out gesture signals that conform to the gesture instruction set and transmit these to signal processor 121 in the second hand device 110 via data communicators 1013 / 1113. It should be noted that a valid gesture data set may be stored in the first hand device 100 (gestures in the valid gesture data set are gesture signals that conform to the gesture instruction set). In addition to storing a valid gesture data set, the second hand device 110 may also store a gesture instruction set. After the signal processor 121 of the second hand device 110 distinguishes the gesture signal and obtains the corresponding output instruction, the output instruction can be sent to the terminal through the data communicator 1113 in the second hand device 110, and the output instruction related to the virtual keyboard is also sent to the keyboard auxiliary device 20. It should be noted that Figure 11 The data communicator in each device can be only one, and communication between multiple devices and within the device can be achieved through functional multiplexing, but it can also be as follows Figure 11 As shown in FIG, multiple data communicators are used to respectively realize data communication between different devices and within the device.

[0221] In the above two examples, the virtual keyboard can be designed as a common planar matrix structure, but it can also be arranged as a spatial spherical matrix structure, for example Figure 13 When the virtual keyboard is a spatial spherical matrix structure, the gesture device can determine the output characters by the spatial position of the palm and the movement of the first finger joint. The keyboard layout and characters corresponding to the left and right hands can be as follows Figure 13 As shown (it should be understood that Figure 13This is for illustration only, and the specific correspondence can be customized by the user.) At this point, the elbows and / or wrists of both hands can be used as the origin of the spherical keyboard. When the palms follow the forearms to raise and lower the elbows and / or wrists to a certain angle with the elbows and / or wrists as the origin, different keyboard rows can be located. When the palms follow the forearms to swing left and right with the elbows and / or wrists as the origin, the keyboard columns controlled by each finger can be located. This makes the user's control more flexible and can adapt to the characteristics of human movement to the greatest extent. It should be noted that Figure 13 The keys in the dotted box are the keys controlled by the left and right hands respectively in half-keyboard mode. The keys outside the dotted box are the keys of the other half of the keyboard extended by the left or right hand in full keyboard mode.

[0222] In the above two examples, the virtual keyboard can be divided into two areas, left and right, as shown in Figure 12 As shown, they correspond to the user's left and right hands for operation respectively. It is particularly important to note that the alphabetical keyboard area on the right hand can be switched to a numeric keyboard or a directional keyboard through commands.

[0223] See the following status table, which shows the status of the gesture device:

[0224] Basic status table

[0225] state illustrate Shutdown Gesture device shutdown Power on Gesture device power on

[0226] Mode State Table

[0227]

[0228]

[0229] Mouse state table

[0230] Elevation mouse movement Move the cursor up, down, left, and right vertically. Planar mouse movement Move the cursor horizontally forward, backward, left, or right.

[0231] Keyboard Status Table

[0232]

[0233] Auxiliary Status Table

[0234] Roll Up Scroll up the page Roll Down Scroll down the page Previous page Flip forward Back page Flip back enlarge Page zoom Zoom out Page zoom out

[0235] See Table 1 below, which shows valid gesture data and its corresponding description according to this embodiment:

[0236] Table 1

[0237]

[0238]

[0239] See Table 2 below, which shows a gesture instruction set according to this embodiment:

[0240] Table 2

[0241]

[0242]

[0243]

[0244] It should be noted that the phrase "After switching states, the relevant initial values ​​are set based on the current palm position" in the table above means that after switching between keyboard and mouse, the current palm position is used as the initial position corresponding to the keyboard's initial row. Furthermore, the aforementioned "left-hand and right-hand state control" refers to switching one hand from keyboard mode to mouse mode while the other hand is in half-keyboard mode.

[0245] In Examples 1 and 2 above, the keyboard assistive device can be used to help beginners of the gesture device confirm the characters output by gestures. The keyboard assistive device receives output instructions related to the virtual keyboard through a wireless connection with the gesture device, and displays the response area and corresponding characters of the virtual keyboard through the display light or bar (i.e., light unit) on the keyboard assistive device. The virtual keyboard can be seen in Table 3 below:

[0246] Table 3

[0247] Serial number Response area or character illustrate 1 Left-hand keyboard area response The red frame of the left keyboard area is bright 2 Left-hand full keyboard area response The red frame of the left and right keyboard areas is bright 3 Right-hand keyboard area response The blue frame of the right keyboard area is bright 4 Right-hand full keyboard area response The blue boxes of the left and right keyboard areas are bright 5 Lowercase keyboard response none 6 Uppercase keyboard response All uppercase character lights are on 7 Right-hand numeric keypad area response The blue frame of the right numeric keypad area is bright 8 Right-hand direction keyboard area response The blue frame of the right-hand direction keyboard area is bright 9 Row Response The row response box lights up 10 Character Response Character response light is on

[0248] According to the gesture device and gesture system provided by the embodiments of the present application, users can operate without having to restrict their hands to a specific plane, and no longer rely on the operating plane constraints of keyboards, mice, etc. This allows users to operate with greater spatial freedom, which is not only beneficial for use in environments unsuitable for traditional keyboards and mice, but also helps prevent computer syndromes such as wrist arthritis. In addition, gesture language can also provide a richer range of keyboard and mouse shortcut commands.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0250] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0251] In addition, the implementation methods described in the various embodiments of the present application can be combined in one embodiment to implement the solution of the present application as long as there is no conflict.

[0252] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0253] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0254] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0255] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A gesture control method, characterized in that: include: Sense hand movements to generate gesture data; Determining, based on the gesture data and a preset gesture instruction set, an output instruction of a simulation input device corresponding to the gesture data in the gesture instruction set; and sending the output instruction to the terminal; The simulated input device includes a virtual keyboard; the virtual keyboard is a spatial spherical matrix structure; the gesture data includes gesture data corresponding to the keyboard keys of the virtual keyboard; the gesture instruction set includes a correspondence between gesture data and instructions, wherein the correspondence includes: gesture data for raising or lowering the left or right hand at an angle of β, and the corresponding instruction is a switching instruction for switching between keyboard rows; β is a preset number greater than 0; Sense hand movements to generate gesture data, including: Using the wrist or elbow as the origin, the angle range of the palm raised and lowered in the vertical plane determines the keyboard row activated by the left or right hand; using the wrist or elbow as the origin, the angle of the palm rotated in the horizontal plane determines the keyboard columns activated by the left or right hand; wherein, the keyboard columns corresponding to each finger are pre-defined; Determining the keyboard key position corresponding to the gesture data according to the keyboard column corresponding to the finger generating the hand movement, and the enabled keyboard row and keyboard column; With the wrist or elbow as the origin, the angle range of the palm raised and lowered on the vertical plane determines the keyboard row activated by the left or right hand, including: With the palm or wrist or elbow as the origin, the angle α between the left / right palm on the vertical plane and the horizontal plane when entering the keyboard mode is taken as the initial angle, and the initial angle α is determined to correspond to the Y0 line; When the left / right hand palm is deflected by an angle of -β relative to α on the vertical plane, it corresponds to row Y-1; when the left / right hand palm is deflected by an angle of β relative to α on the vertical plane, it corresponds to row Y1; when the left / right hand palm is deflected by an angle of 2β relative to α on the vertical plane, it corresponds to row Y2; when the left / right hand palm is deflected by an angle of β relative to α on the vertical plane, it corresponds to row Y3; The Y-1 line is the last row of the virtual keyboard, the Y0 line is the second to last row of the virtual keyboard, the Y1 line is the third to last row of the virtual keyboard, the Y2 line is the fourth to last row of the virtual keyboard, and the Y3 line is the fifth to last row of the virtual keyboard; The simulated input device also includes a virtual mouse; when switching from mouse mode to keyboard mode, the current position of the palm is used as the initial position to locate the Y0 row of the virtual keyboard.

2. The gesture control method according to claim 1, wherein: Determining, based on the gesture data and a preset gesture instruction set, the output instruction of the simulated input device corresponding to the gesture data in the gesture instruction set includes: Processing the gesture data to parse out a gesture signal that complies with the gesture instruction set; An output instruction corresponding to the gesture signal in the gesture instruction set is determined according to the current input mode of the analog input device.

3. The gesture control method according to claim 1, wherein: After determining the output instruction of the simulated input device corresponding to the gesture data in the gesture instruction set according to the gesture data and the preset gesture instruction set, the method further includes: An output instruction related to the virtual keyboard is sent to a keyboard auxiliary device, so that the keyboard auxiliary device emits light in a specific area according to the received output instruction to prompt a response area or character of the virtual keyboard.

4. The gesture control method according to claim 3, wherein: The correspondence between the response area or character of the virtual keyboard and the luminous area on the keyboard auxiliary device includes at least one of the following: The left-hand keyboard area of ​​the virtual keyboard responds, and the left-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up; The left-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard areas of the corresponding keyboard assistive device light up; The right-hand keyboard area of ​​the virtual keyboard responds, and the right-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up; The right-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard areas of the corresponding keyboard assistive device light up; The uppercase keyboard of the virtual keyboard responds, and the uppercase character lights of the corresponding keyboard auxiliary device are all lit; The right-hand numeric keypad area of ​​the virtual keyboard responds, and the right-hand numeric keypad area of ​​the corresponding keyboard auxiliary device lights up; The right-hand direction keyboard area of ​​the virtual keyboard responds, and the right-hand direction keyboard area of ​​the corresponding keyboard auxiliary device lights up; If a row of the virtual keyboard responds, the row of the corresponding keyboard auxiliary device lights up; When a character on the virtual keyboard is responded, the character on the corresponding keyboard auxiliary device is lit; The lowercase keyboard of the virtual keyboard responds, and the relevant area of ​​the corresponding keyboard auxiliary device is not lit.

5. The gesture control method according to any one of claims 1 to 4, wherein: The corresponding relationship also includes at least one of the following: The corresponding instruction for the gesture data of hand A making a fist and then releasing it is: the switching instruction for hand A's keyboard mode to hand A's mouse mode; The corresponding instruction for the gesture data of hand B making a fist and then releasing it is: the instruction for switching hand B's keyboard mode to hand B's mouse mode; The gesture data of rotating the left palm by n degrees (n is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switching instruction between an all-lowercase keyboard mode and an all-uppercase keyboard mode, and / or a switching instruction between a numeric keyboard mode and a directional keyboard mode; The gesture data of rotating the right palm by m1 degrees (m1 is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switch instruction between the right-hand uppercase keyboard mode and the directional keyboard mode, and / or a switch instruction between the right-hand lowercase keyboard mode and the numeric keyboard mode; The gesture data of hand B's palm rotating m2 degrees (m2 is a preset number greater than 0) and then returning to its original position corresponds to the instruction for switching between the right-handed plane mouse mode and the right-handed vertical mouse mode; The gesture data of hand B being raised corresponds to the command for switching between the flat mouse mode and the precise positioning mouse mode for hand B. The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise positioning mouse mode to the flat mouse mode; The gesture data of hand B moving forward, backward, left, and right corresponds to the following instructions: the movement instructions of hand B's flat mouse moving forward, backward, left, and right; The gesture data of hand B being raised corresponds to the instruction for switching hand B's flat mouse mode to precise flat mouse mode. The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise flat mouse mode to the flat mouse mode; The gesture data of hand B moving up, down, left, and right corresponds to the movement instructions of the vertical mouse of hand B moving up, down, left, and right; The gesture data of hand B moving forward corresponds to the instruction for switching hand B's vertical mouse mode to precise vertical mouse mode. The gesture data of hand B moving back corresponds to the instruction for switching hand B's precise vertical mouse mode to vertical mouse mode. The gesture data of hand A moving up, down, left, right, forward, and backward corresponds to the following commands: the mouse of hand A assists in scrolling up, scrolling down, turning the page forward, turning the page backward, zooming in, and zooming out; The gesture data of the back contact between hand A and hand B corresponds to the instruction to power on or off the device. In the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; in the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

6. The gesture control method according to any one of claims 1 to 4, wherein: The gesture control method further includes: determining a control instruction corresponding to the current input mode of the analog input device according to the current input mode of the analog input device and a preset correspondence between the input mode and the control instruction, and sending the control instruction to the terminal.

7. The gesture control method according to claim 6, wherein: The simulated input device also includes a virtual mouse; The corresponding relationship between the input mode and the control instruction includes at least one of the following: When the current input mode is that hand A switches from keyboard mode to mouse mode and hand B is in half-keyboard mode, the control instruction is: a switching instruction for hand B half-keyboard mode to hand B full-keyboard mode; When the current input mode is that hand A switches from mouse mode to keyboard mode and hand B is in full keyboard mode, the control instruction is: a switching instruction for hand B's full keyboard mode to hand B's half keyboard mode; When the current input mode is that hand B switches from keyboard mode to mouse mode and hand A is in half-keyboard mode, the control instruction is: a switching instruction for hand A's half-keyboard mode to hand A's full-keyboard mode; When the current input mode is that hand B switches from mouse mode to keyboard mode and hand A is in full keyboard mode, the control instruction is: a switching instruction for hand A's full keyboard mode to hand A's half keyboard mode; In the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; In the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

8. A gesture device, characterized in that: include: First-hand equipment, The first hand device includes a processor and a sensor; The sensor is used to sense hand movements and generate gesture data; The processor is configured to obtain gesture data generated by the sensor, determine an output instruction of the analog input device corresponding to the gesture data in the gesture instruction set based on the gesture data and a preset gesture instruction set, and send the output instruction to an external terminal; The simulated input device includes a virtual keyboard; the virtual keyboard is a spatial spherical matrix structure; the gesture data includes gesture data corresponding to the keyboard keys of the virtual keyboard; the gesture instruction set includes a correspondence between gesture data and instructions, wherein the correspondence includes: gesture data for raising or lowering the left or right hand by an angle β, and the corresponding instruction is: a switching instruction for switching between keyboard rows; β is a preset number greater than 0; The sensor is specifically configured to: determine the keyboard row activated by the left or right hand based on the angle range of the palm raised and lowered on a vertical plane, with the wrist or elbow as the origin; determine the keyboard columns activated by the left or right hand based on the angle of rotation of the palm on a horizontal plane, with the wrist or elbow as the origin; wherein the keyboard columns corresponding to the respective fingers are predefined; and determine the keyboard key position corresponding to the gesture data based on the keyboard column corresponding to the finger where the sensor that senses the hand movement is located, as well as the activated keyboard row and keyboard column; The wrist or elbow is used as the origin, and the angle range of the palm raised and lowered on the vertical plane is used to determine the keyboard row activated by the left or right hand, including: With the palm or wrist or elbow as the origin, the angle α between the left / right palm on the vertical plane and the horizontal plane when entering the keyboard mode is taken as the initial angle, and the initial angle α is determined to correspond to the Y0 line; When the left / right hand palm is deflected by an angle of -β relative to α on the vertical plane, it corresponds to row Y-1; when the left / right hand palm is deflected by an angle of β relative to α on the vertical plane, it corresponds to row Y1; when the left / right hand palm is deflected by an angle of 2β relative to α on the vertical plane, it corresponds to row Y2; when the left / right hand palm is deflected by an angle of β relative to α on the vertical plane, it corresponds to row Y3; The Y-1 line is the last row of the virtual keyboard, the Y0 line is the second to last row of the virtual keyboard, the Y1 line is the third to last row of the virtual keyboard, the Y2 line is the fourth to last row of the virtual keyboard, and the Y3 line is the fifth to last row of the virtual keyboard; The analog input device further includes a virtual mouse; the sensor is further configured to: when switching from mouse mode to keyboard mode, locate the Y0 row of the virtual keyboard using the current position of the palm as the initial position.

9. The gesture device according to claim 8, wherein: The gesture device further includes a second hand device and a signal connector; The second hand device includes a processor and a sensor; the processors of the first hand device and the second hand device are respectively communicatively connected to the signal connector, and send the output instruction to the external terminal through the signal connector.

10. The gesture device according to claim 9, wherein: The processors of the first hand device and the second hand device each include an acquisition circuit, a motion recognition circuit, and a data communicator; the signal connector includes a signal processor; The acquisition circuit collects gesture data from the sensor and sends it to the action recognition circuit; the action recognition circuit processes the gesture data, parses out a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor via the data communicator; The signal processor determines an output instruction corresponding to the received gesture signal in the gesture instruction set according to the current state of the analog input device, and sends the output instruction to the external terminal.

11. The gesture device according to claim 8, wherein: The gesture device further includes a second hand device; the second hand device includes a processor, a sensor, and a first data communicator; The processor of the first hand device is communicatively connected to the processor of the second hand device, and the processor of the first hand device communicates with the external terminal through the first data communicator.

12. The gesture device according to claim 11, wherein: The processors of the first hand device and the second hand device each include an acquisition circuit, a motion recognition circuit, and a second data communicator; the second hand device also includes a signal processor; The acquisition circuit collects gesture data from the sensor and sends it to the action recognition circuit; the action recognition circuit processes the gesture data, parses out a gesture signal that conforms to the gesture instruction set, and sends the gesture signal to the signal processor via the second data communicator; The signal processor determines an output instruction corresponding to the received gesture signal in the gesture instruction set according to the current state of the analog input device, and sends the output instruction to the external terminal through the first data communicator.

13. The gesture device according to claim 8, wherein: The sensor includes a finger sensor; the finger sensor can be arranged at a first knuckle and / or a second knuckle of a finger.

14. The gesture device according to claim 13, wherein: The sensor further includes a palm sensor; the palm sensor is arranged in the processor for sensing palm movements to generate gesture data.

15. The gesture device according to any one of claims 9 to 12, wherein: Also included is a switch module; The switch module includes: a first power switch, a first Hall sensor, and a first magnetic block provided in the first hand device, and a second power switch, a second Hall sensor, and a second magnetic block provided in the second hand device; When the first hand device and the second hand device approach and separate, the magnetic fields of the first magnetic block and the second magnetic block change, causing the first Hall sensor and the second Hall sensor to generate current and drive the first power switch and the second power switch to turn on or off.

16. The gesture device according to any one of claims 8 to 14, wherein: The simulated input device further includes a virtual mouse, and the corresponding relationship further includes at least one of the following: The corresponding instruction for the gesture data of hand A making a fist and then releasing it is: the switching instruction for hand A's keyboard mode to hand A's mouse mode; The corresponding instruction for the gesture data of hand B making a fist and then releasing it is: the instruction for switching hand B's keyboard mode to hand B's mouse mode; The gesture data of rotating the left palm by n degrees (n is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switch instruction between an all-lowercase keyboard mode and an all-uppercase keyboard mode, and / or a switch instruction between a right-hand numeric keyboard mode and a directional keyboard mode; The gesture data of rotating the right palm by m1 degrees (m1 is a preset number greater than 0) and then returning to the original position corresponds to the following instructions: a switch instruction between the right-hand uppercase keyboard mode and the directional keyboard mode, and / or a switch instruction between the right-hand lowercase keyboard mode and the numeric keyboard mode; The gesture data of hand B's palm rotating m2 degrees (m2 is a preset number greater than 0) and then returning to its original position corresponds to the instruction for switching between the right-handed plane mouse mode and the right-handed vertical mouse mode; The gesture data of hand B being raised corresponds to the command for switching between the flat mouse mode and the precise positioning mouse mode for hand B. The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise positioning mouse mode to the flat mouse mode; The gesture data of hand B moving forward, backward, left, and right corresponds to the following instructions: the movement instructions of hand B's flat mouse moving forward, backward, left, and right; The gesture data of hand B being raised corresponds to the instruction for switching hand B's flat mouse mode to precise flat mouse mode. The corresponding instruction for the B hand's lowering gesture data is: the instruction for switching the B hand's precise flat mouse mode to the flat mouse mode; The gesture data of hand B moving up, down, left, and right corresponds to the movement instructions of the vertical mouse of hand B moving up, down, left, and right; The gesture data of hand B moving forward corresponds to the instruction for switching hand B's vertical mouse mode to precise vertical mouse mode. The gesture data of hand B moving back corresponds to the instruction for switching hand B's precise vertical mouse mode to vertical mouse mode. The gesture data of hand A moving up, down, left, right, forward, and backward corresponds to the following commands: the mouse of hand A assists in scrolling up, scrolling down, turning the page forward, turning the page backward, zooming in, and zooming out; The gesture data of the back contact between hand A and hand B corresponds to the instruction to power on or off the device. In the keyboard mode, the virtual keyboard is divided into a left-hand keyboard area and a right-hand keyboard area; in the mouse mode, hand A is the left hand and hand B is the right hand, or hand A is the right hand and hand B is the left hand.

17. A gesture system, characterized in that: comprising a keyboard auxiliary device and a gesture device according to any one of claims 8 to 16; The keyboard auxiliary device is communicatively connected to the gesture device and receives output instructions related to the virtual keyboard sent by the gesture device; The keyboard auxiliary device is provided with a processor and a light emitting unit; When the processor receives an output instruction related to the virtual keyboard sent by the gesture device, it controls the light-emitting unit to emit light in a specific area according to the output instruction to display a response area or character of the virtual keyboard.

18. The gesture system of claim 17, wherein: The keyboard auxiliary device is a physical style of a preset virtual keyboard; The correspondence between the response area or character of the virtual keyboard and the light-emitting unit on the keyboard auxiliary device includes at least one of the following: The left-hand keyboard area of ​​the virtual keyboard responds, and the left-hand keyboard area light-emitting unit of the corresponding keyboard auxiliary device lights up; The left-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard area light-emitting units of the corresponding keyboard auxiliary device light up; The right-hand keyboard area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand keyboard area of ​​the corresponding keyboard auxiliary device lights up; The right-hand full keyboard area of ​​the virtual keyboard responds, and the left-hand and right-hand keyboard area light-emitting units of the corresponding keyboard auxiliary device light up; The uppercase keyboard of the virtual keyboard responds, and the uppercase character lights of the corresponding keyboard auxiliary device are all lit; The right-hand numeric keypad area of ​​the virtual keyboard responds, and the light-emitting unit of the right-hand numeric keypad area of ​​the corresponding keyboard auxiliary device lights up; The right-hand direction keyboard area of ​​the virtual keyboard responds, and the right-hand direction keyboard area light-emitting unit of the corresponding keyboard auxiliary device lights up; When a row of the virtual keyboard responds, the corresponding light-emitting unit of the keyboard auxiliary device lights up; When a character on the virtual keyboard is responded, the corresponding light-emitting unit of the character on the keyboard auxiliary device lights up; The lowercase keyboard of the virtual keyboard responds, and the corresponding light-emitting unit in the relevant area of ​​the keyboard auxiliary device is off.

Images