Multi-sensor fusion gesture recognition device and method

Abstract

The invention discloses a multi-sensor fusion gesture recognition device and method, and relates to the technical field of man-machine interaction, and the method comprises the steps: obtaining original data which comprises quaternion and flexion and extension angles; the original data are calculated and processed to obtain hand postures, and the hand postures comprise the roll angle of the hand back, the pitch angle of the hand back, the course angle of the hand back and the flexion and extension angles of four fingers; determining gesture dynamic and static states according to the hand posture and a preset dynamic and static state distinguishing mode; and generating a control instruction according to the hand posture, the gesture dynamic and static state and a preset control instruction generation mode. According to the gesture recognition device and method based on multi-sensor fusion, the hand posture can be detected and recognized, the dynamic state and the static state of the gesture can be distinguished and the control instruction can be generated according to the hand posture recognition result, and the problem of gesture recognition stability under any posture can be solved; and the distinguishing problem of dynamic and static states and the generation problem of a control instruction are solved, and the control requirement of actual controlled equipment is met.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multi-sensor fusion gesture recognition device and method. Background Art

[0002] With the development of human-computer interaction technology, the interaction between the virtual world and the real world is becoming closer and closer. Among them, gesture interaction is the most widely used technology in human-computer interaction. Traditional gesture interaction is mainly based on machine vision. However, machine vision requires a huge image acquisition and processing platform as support and cannot be applied to devices with small platforms.

[0003] An inertial sensor (IMU) is a sensor that can measure the acceleration and angular velocity of an object, usually composed of an accelerometer and a gyroscope. By measuring and analyzing the motion state of an object, an inertial sensor can achieve real-time monitoring and recognition of the object's posture and movement. Gesture interaction technology based on inertial sensors uses the data from these sensors and combines algorithms to recognize and analyze user gestures, thereby achieving interaction with computers or other devices. This interaction method has the advantages of high recognition accuracy, fast feedback speed, and easy use. It can meet users' needs for more intuitive and natural interaction methods and is suitable for a variety of application scenarios such as virtual reality, augmented reality, somatosensory interaction, and smart wearable devices.

[0004] However, the existing gesture interaction technology based on inertial sensors has a stability bottleneck. When using inertial sensors to solve hand posture problems, the stability is poor, and it can only recognize hand postures. It does not consider how to distinguish between dynamic and static states and generate control instructions based on hand postures, and cannot be directly deployed to the product end to realize product control. Summary of the invention

[0005] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a multi-sensor fusion gesture recognition device and method, which solves the problem of gesture recognition stability under arbitrary postures, and also solves the problem of distinguishing between dynamic and static states and the problem of generating control instructions, and can meet the control requirements of the actual controlled equipment.

[0006] The technical solution of the present invention is as follows:

[0007] In a first aspect, a multi-sensor fusion gesture recognition device is provided, the device comprising: a glove body, an inertial sensor, a flexible sensor, a data processing unit and a user interface;

[0008] The glove body is used to be worn on the user's hand and to fix the inertial sensor, the flexible sensor, the data processing unit and the user interface;

[0009] The inertial sensor, the flexible sensor and the user interface are respectively connected to the data processing unit;

[0010] The inertial sensor is arranged at the back of the glove body, and is used to collect and send quaternions, acceleration and angular velocity to the data processing unit;

[0011] The flexible sensors are respectively arranged at the little finger, ring finger, middle finger and index finger of the glove body, and are used to collect and send the flexion and extension angles of the fingers to the data processing unit;

[0012] The data processing unit is used to receive and process the quaternion, the acceleration, the angular velocity and the flexion and extension angle, obtain the hand posture, determine the dynamic and static state of the gesture according to the hand posture and a preset dynamic and static state distinguishing method, and generate a control instruction according to the hand posture, the dynamic and static state of the gesture and a preset control instruction generating method;

[0013] The user interface can be connected to an external device for data interaction.

[0014] In a second aspect, a gesture recognition method is provided, the method being applied to the multi-sensor fusion gesture recognition device, the method comprising:

[0015] Acquiring original data, wherein the original data includes: quaternion and flexion and extension angle;

[0016] Calculating and processing the raw data to obtain the hand posture, wherein the hand posture includes: a roll angle of the back of the hand, a pitch angle of the back of the hand, a heading angle of the back of the hand, and flexion and extension angles of four fingers;

[0017] Determining the dynamic and static state of the gesture according to the hand posture and a preset dynamic and static state distinguishing method;

[0018] A control instruction is generated according to the hand posture, the dynamic and static state of the gesture, and a preset control instruction generation method.

[0019] In some optional implementations, obtaining raw data includes:

[0020] Initializing the gesture recognition device;

[0021] Using the inertial sensor, collecting and acquiring the quaternion;

[0022] The flexion and extension angle is acquired by utilizing the flexible sensor.

[0023] In some optional implementations, the calculating and processing the raw data to obtain the hand posture includes:

[0024] Normalizing the quaternion to obtain a normalized quaternion;

[0025] Based on the angle solution model, the normalized quaternion is calculated and processed to obtain the roll angle, pitch angle and heading angle of the back of the hand;

[0026] The flexion and extension angle corresponding to each finger is determined according to the acquired flexion and extension angle.

[0027] In some optional implementations, the normalized expression is:

[0028]

[0029] in, Represents the normalized quaternion i+1th element component, i=0,1,2,3,q i Represents the i+1th element component of the quaternion, q0 represents the 1st element component of the quaternion, q1 represents the 2nd element component of the quaternion, q2 represents the 3rd element component of the quaternion, and q3 represents the 4th element component of the quaternion.

[0030] In some optional implementations, the angle calculation model expression is:

[0031]

[0032] in, represents the roll angle of the back of the hand, arctan represents the calculation of the inverse tangent trigonometric function, θ represents the pitch angle of the back of the hand, arcsin represents the calculation of the inverse sine trigonometric function, and ψ represents the heading angle of the back of the hand.

[0033] In some optional implementations, the preset dynamic and static state distinction method includes:

[0034] When the roll angle of the back of the hand is within a first preset angle range, it is determined that the gesture is in a static state;

[0035] When the roll angle of the back of the hand is within a second preset angle range, the gesture is determined to be in a dynamic state;

[0036] The first preset angle range is [120 0 ,180 0 ), the second preset angle range is (-180 0 ,120 0 ).

[0037] In some optional implementations, generating a control instruction according to the hand posture, the dynamic and static state of the gesture, and a preset control instruction generation method includes:

[0038] When the gesture is in a static state, the flexion and extension angles of two specified fingers are selected as control instruction identifiers, and an acceleration control instruction containing a corresponding speed is generated according to the flexion and extension angle of one of the specified fingers. A deceleration control instruction containing a corresponding speed is generated according to the flexion and extension angle of the other specified finger. The corresponding speed is obtained by mapping the corresponding flexion and extension angle through a first linear mapping model.

[0039] In some optional implementations, the generating of the control instruction according to the hand posture, the dynamic and static state of the gesture and the preset control instruction generating method further includes:

[0040] When the gesture is in a dynamic state, the flexion and extension angle is processed to determine whether there is a faulty flexible sensor, and if so, the flexion and extension angle data collected by the faulty flexible sensor is eliminated;

[0041] The roll angle of the back of the hand, the pitch angle of the back of the hand, the heading angle of the back of the hand, and the average of the flexion and extension angles of the fingers are selected as control instruction identifiers, and dynamic control instructions are generated according to the average of the roll angle, pitch angle, heading angle and flexion and extension angle. The dynamic control instructions include: forward action instructions, backward action instructions, ascending action instructions, descending action instructions, left turn action instructions, right turn action instructions and hovering action instructions.

[0042] In some optional implementations, generating a dynamic control instruction according to the average of the roll angle, the pitch angle, the heading angle, and the flexion and extension angle includes:

[0043] If the rolling angle is within the range of 90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a forward motion command is generated;

[0044] If the rolling angle is within the range of -90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a backward motion command is generated;

[0045] If the pitch angle changes, and the average value of the flexion and extension angle is between 0° and m°, an ascending action command or a descending action command is generated according to the change value of the pitch angle;

[0046] If the heading angle changes and the mean value of the flexion and extension angle is within 90°±m°, a left turn action command or a right turn action command is generated according to the change value of the heading angle;

[0047] If the roll angle is within ±m° and the mean flexion and extension angle is within 90°±m°, a hovering action command is generated;

[0048] Wherein, m represents a preset error parameter.

[0049] The main advantages of the technical solution of the present invention are as follows:

[0050] The multi-sensor fusion gesture recognition device and method of the present invention can detect and identify hand gestures, and distinguish the dynamic and static states of gestures and generate control instructions based on the hand gesture recognition results. It can solve the problem of gesture recognition stability under arbitrary postures, as well as the problem of distinguishing dynamic and static states and generating control instructions, and meet the control requirements of the actual controlled equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0052] Figure 1 It is a schematic diagram of the composition of a multi-sensor fusion gesture recognition device disclosed in an embodiment of the present invention, wherein the glove body is not shown;

[0053] Figure 2 A schematic diagram of the positions of an inertial sensor and a flexible sensor of a gesture recognition device using multi-sensor fusion disclosed in an embodiment of the present invention;

[0054] Figure 3 The present invention is a flowchart of a gesture recognition method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] The present invention discloses a multi-sensor fusion gesture recognition device and method, which solves the problem of gesture recognition stability under any posture, and also solves the problem of distinguishing between dynamic and static states and the problem of generating control instructions, and can meet the control requirements of the actual controlled equipment. The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0059] Embodiment 1

[0060] refer to Figure 1-2 , Figure 1 The present invention is a schematic diagram of a multi-sensor fusion gesture recognition device disclosed in an embodiment of the present invention. Figure 2 The schematic diagram of the positions of the inertial sensor and the flexible sensor of a multi-sensor fusion gesture recognition device disclosed in an embodiment of the present invention. To solve the above technical problems, an embodiment of the present invention provides a multi-sensor fusion gesture recognition device, which includes: a glove body, an inertial sensor 100, a flexible sensor 200, a data processing unit 300 and a user interface 400;

[0061] The glove body is used to be worn on the user's hand and to fix the inertial sensor 100, the flexible sensor 200, the data processing unit 300 and the user interface 400;

[0062] The inertial sensor 100 , the flexible sensor 200 , and the user interface 400 are respectively connected to the data processing unit 300 ;

[0063] The inertial sensor 100 is disposed on the back of the glove body and is used to collect and send quaternions, acceleration and angular velocity to the data processing unit 300;

[0064] Flexible sensors 200 are respectively arranged at the little finger, ring finger, middle finger and index finger of the glove body, and are used to collect and send the flexion and extension angles of the fingers to the data processing unit 300;

[0065] The data processing unit 300 is used to receive and process quaternions, accelerations, angular velocities, and flexion and extension angles to obtain hand postures, to determine the dynamic and static states of gestures according to the hand postures and a preset dynamic and static state distinction method, and to generate control instructions according to the hand postures, the dynamic and static states of gestures, and a preset control instruction generation method;

[0066] The user interface 400 can be connected to an external device for data interaction.

[0067] It should be noted that the glove body Figure 1 Not shown.

[0068] It should be noted that when the inertial sensor 100 is set on the back of the hand, the X-axis of the inertial sensor 100 is parallel to the back of the hand and points to the direction of the fingers, the Y-axis of the inertial sensor 100 is parallel to the back of the hand and points to the outside of the back of the hand, and the Z-axis of the inertial sensor 100 is perpendicular to the back of the hand and points upward.

[0069] In the embodiment of the present invention, the user interface 400 can be connected to an external device to achieve data interaction between the device and the external device. Optionally, the user interface 400 can be a physical interface or a standard wireless Bluetooth or WiFi communication unit.

[0070] In the embodiment of the present invention, the data processing unit 300 uses the user interface 400 as a channel to perform data interaction with the connected external device, and the data interaction includes: receiving data input by the external device, and outputting hand gestures and control command information to the external device.

[0071] It should be noted that the multi-sensor fusion gesture recognition device provided in the embodiment of the present invention can be applied to a wearable human-computer interaction system and can be connected to a local server or cloud server for human-computer interaction management, etc., which is not limited in the embodiment of the present invention.

[0072] In an embodiment of the present invention, the data processing unit 300 may be a CPU; the data processing unit 300 may be respectively connected to the above-mentioned inertial sensor 100, flexible sensor 200 and user interface 400 through a communication bus; the data processing unit 300 may be provided with or connected to a memory, and the memory may be used to store hand posture, gesture dynamic and static states and control instruction information obtained by the data processing unit 300; wherein the memory may be a high-speed RAM memory or a ROM memory.

[0073] The multi-sensor fusion gesture recognition device provided in the embodiment of the present invention sets sensors to respectively detect the motion information of the back of the hand and the motion information of the fingers, recognizes the hand posture according to the detection information, and distinguishes the dynamic and static states of the gesture and generates control instructions according to the hand posture recognition result. It can solve the problem of gesture recognition stability under arbitrary postures, as well as the problem of distinguishing the dynamic and static states and the problem of generating control instructions, and meet the control requirements of the actual controlled equipment.

[0074] Embodiment 2

[0075] refer to Figure 3 , Figure 3The flowchart of a gesture recognition method disclosed in an embodiment of the present invention is shown in FIG. To solve the above technical problem, an embodiment of the present invention provides a gesture recognition method, which can be applied to the multi-sensor fusion gesture recognition device described in the first embodiment.

[0076] like Figure 3 As shown, the gesture recognition method provided by the embodiment of the present invention may include the following steps 1 to 4:

[0077] Step 1, obtaining original data;

[0078] In the embodiment of the present invention, the original data includes: quaternion and flexion and extension angle.

[0079] Step 2: Calculate and process the original data to obtain the hand posture;

[0080] In the embodiment of the present invention, the hand posture includes: a roll angle of the back of the hand, a pitch angle of the back of the hand, a heading angle of the back of the hand, and flexion and extension angles of four fingers.

[0081] Step 3, determining the dynamic and static state of the gesture according to the hand posture and a preset dynamic and static state distinction method;

[0082] It should be noted that the preset dynamic and static state distinction method is set according to the control requirements of the actual controlled equipment and is not specifically limited here.

[0083] Step 4, generating a control instruction according to the hand posture, the dynamic and static state of the gesture, and a preset control instruction generation method;

[0084] It should be noted that the preset control instruction generation method is set according to the control requirements of the actual controlled equipment and is not specifically limited here.

[0085] By implementing the gesture recognition method provided by the embodiment of the present invention in the multi-sensor fusion gesture recognition device described in the first embodiment above, the problem of gesture recognition stability under any posture can be solved, as well as the problem of distinguishing between dynamic and static states and the problem of generating control instructions, thereby meeting the control requirements of the actual controlled equipment.

[0086] Further, in an optional implementation of the embodiment of the present invention, in the above step 1, the obtaining of raw data includes the following steps 101 to 103:

[0087] Step 101, initializing the gesture recognition device;

[0088] In an embodiment of the present invention, initializing the gesture recognition device includes: turning on the gesture recognition device, making the palm of the glove face downward, and the back of the hand parallel to the ground, and maintaining the designated state for 5-10 seconds. The designated state may be a "clenched fist" state or a "fully extended fingers" state.

[0089] Step 102, using an inertial sensor to collect and obtain quaternions;

[0090] In the embodiment of the present invention, the inertial sensor is disposed at the back of the hand to collect quaternions, acceleration and angular velocity at the back of the hand.

[0091] The acceleration is the XYZ three-axis acceleration measured by the inertial sensor, and the angular velocity is the XYZ three-axis angular velocity measured by the inertial sensor.

[0092] It should be noted that when the inertial sensor is set on the back of the hand, the X-axis of the inertial sensor is parallel to the back of the hand and points to the direction of the fingers, the Y-axis of the inertial sensor is parallel to the back of the hand and points to the outside of the back of the hand, and the Z-axis of the inertial sensor is perpendicular to the back of the hand and points upward.

[0093] Step 103: using a flexible sensor to collect and obtain the flexion and extension angle.

[0094] In an embodiment of the present invention, each finger except the thumb is provided with a flexible sensor, that is, the little finger, ring finger, middle finger and index finger are all provided with flexible sensors, and the flexible sensor provided at each finger is used to collect the flexion and extension angle of the finger.

[0095] In the embodiment of the present invention, the flexion and extension angle of the finger is obtained by mapping the voltage value of the flexible sensor arranged at the finger into an angle.

[0096] Specifically, the angle mapping formula is expressed as:

[0097]

[0098] Among them, α represents the flexion and extension angle of the finger, U0 represents the voltage value output by the flexible sensor at the current moment, and U represents the maximum voltage value that the flexible sensor can output.

[0099] In the embodiment of the present invention, by powering on and initializing the gesture recognition device before collecting and acquiring the quaternion and the flexion and extension angle, the reliability of the collected quaternion and the flexion and extension angle data can be ensured.

[0100] In another optional implementation of the embodiment of the present invention, in the above step 1, the obtaining of raw data includes the following steps 111 to 116:

[0101] Step 111, using an inertial sensor to collect and obtain the three-axis angular velocity at time t;

[0102] In the embodiment of the present invention, the three-axis angular velocity at time t is acquired by using an inertial sensor disposed at the back of the hand.

[0103] Step 112, using a flexible sensor to collect the flexion and extension angle at time t;

[0104] In the embodiment of the present invention, the flexion and extension angles of the four fingers at time t are collected by using flexible sensors arranged at the fingers.

[0105] Step 113, based on the combined angular velocity calculation model, the three-axis angular velocity is calculated and processed to obtain the combined angular velocity of the back of the hand;

[0106] In the embodiment of the present invention, the angular velocity calculation model expression is:

[0107]

[0108] Among them, g st (t) represents the angular velocity of the back of the hand at time t, g x (t) represents the X-axis component of the three-axis angular velocity at time t, g y (t) represents the Y-axis component of the three-axis angular velocity at time t, g z (t) represents the Z-axis component of the three-axis angular velocity at time t, and t represents the sampling time.

[0109] Step 114, based on the flexion and extension angle combined change value calculation model, the flexion and extension angle is calculated and processed to obtain the finger flexion and extension angle combined change value;

[0110] In the embodiment of the present invention, the calculation model expression of the flexion-extension angle change value is:

[0111]

[0112] Among them, Δα(t) represents the change value of the finger flexion and extension angle at time t, represents the flexion and extension angle of the ith finger at time t, represents the flexion and extension angle of the i-th finger at time t-1, where time t-1 is a sampling time before time t.

[0113] It should be noted that, when i=1, the i-th finger represents the little finger; when i=2, the i-th finger represents the ring finger; when i=3, the i-th finger represents the middle finger; when i=4, the i-th finger represents the index finger.

[0114] Step 115, determining whether the hand back angular velocity and the finger flexion and extension angle combined change value are respectively less than the angular velocity threshold and the flexion and extension angle combined change threshold, and obtaining a determination result;

[0115] In the embodiment of the present invention, the angular velocity threshold and the flexion-extension angle change threshold are used to determine whether the gesture recognition device is in a stationary state. It should be noted that the stationary state does not necessarily mean being motionless. When the movement amplitude of the gesture recognition device is small and negligible, the gesture recognition device is also regarded as being in a stationary state.

[0116] Therefore, in the embodiment of the present invention, the specific values ​​of the angular velocity threshold and the flexion-extension angle change threshold are set according to the actual situation, but the values ​​cannot be too large. For example, the angular velocity threshold is a value within the range of 5-6° / s, and the flexion-extension angle change threshold is a value within the range of 5-6°.

[0117] Step 116, when the judgment result is yes, execute steps 117 to 118;

[0118] When the judgment result is no, execute step 111;

[0119] Step 117, using an inertial sensor to collect and obtain quaternions;

[0120] In the embodiment of the present invention, the inertial sensor is disposed at the back of the hand, and can collect the quaternion, acceleration and angular velocity at the back of the hand.

[0121] Step 118: Utilize the flexible sensor to collect and obtain the flexion and extension angle.

[0122] In an embodiment of the present invention, each finger except the thumb is provided with a flexible sensor, that is, the little finger, ring finger, middle finger and index finger are all provided with flexible sensors, and the flexible sensor provided at each finger is used to collect the flexion and extension angle of the finger.

[0123] In the embodiment of the present invention, the flexion and extension angle of the finger is obtained by mapping the voltage value of the flexible sensor arranged at the finger into an angle.

[0124] Specifically, the angle mapping formula is expressed as:

[0125]

[0126] Among them, α represents the flexion and extension angle of the finger, U0 represents the voltage value output by the flexible sensor at the current moment, and U represents the maximum voltage value that the flexible sensor can output.

[0127] In the embodiment of the present invention, by ensuring that the gesture recognition device is in a stationary state before collecting and acquiring the quaternion and the flexion and extension angle, the reliability of the collected quaternion and the flexion and extension angle data can be guaranteed.

[0128] Further, in an optional implementation of the embodiment of the present invention, in the above step 2, the raw data is calculated and processed to obtain the hand posture, including the following steps 21-23:

[0129] Step 21, normalizing the quaternion to obtain a normalized quaternion;

[0130] In the embodiment of the present invention, the normalization expression is:

[0131]

[0132] in, Represents the normalized quaternion i+1th element component, i=0,1,2,3,q i Represents the i+1th element component of the quaternion, q0 represents the 1st element component of the quaternion, q1 represents the 2nd element component of the quaternion, q2 represents the 3rd element component of the quaternion, and q3 represents the 4th element component of the quaternion.

[0133] Step 22, based on the angle solution model, the normalized quaternion is calculated and processed to obtain the roll angle, pitch angle and heading angle of the back of the hand;

[0134] In the embodiment of the present invention, the angle calculation model expression is:

[0135]

[0136] in, represents the roll angle of the back of the hand, arctan represents the calculation of the inverse tangent trigonometric function, θ represents the pitch angle of the back of the hand, arcsin represents the calculation of the inverse sine trigonometric function, and ψ represents the heading angle of the back of the hand.

[0137] Step 23, determining the flexion and extension angle corresponding to each finger according to the obtained flexion and extension angle;

[0138] In an embodiment of the present invention, each finger except the thumb is provided with a flexible sensor, that is, the little finger, ring finger, middle finger and index finger are all provided with flexible sensors, and the flexible sensor provided at each finger is used to collect the flexion and extension angle of the finger.

[0139] In the embodiment of the present invention, the flexion and extension angle corresponding to each finger is directly determined according to the flexion and extension angle collected by the flexible sensor arranged at the finger.

[0140] Further, in an optional implementation of the embodiment of the present invention, the preset dynamic and static state distinction method includes:

[0141] When the roll angle of the back of the hand is within a first preset angle range, it is determined that the gesture is in a static state;

[0142] When the roll angle of the back of the hand is within a second preset angle range, the gesture is determined to be in a dynamic state;

[0143] In order to facilitate the operation, the first preset angle range is set to [120 0 ,180 0 ), the second preset angle range is set to (-180 0 ,120 0 ).

[0144] Specifically, when the gesture recognition device is used for controlling a drone, that is, the actual controlled equipment is a drone, the preset dynamic and static state distinction method can adopt the above-defined method.

[0145] Further, based on the preset static and dynamic state distinguishing method set above, determining the static and dynamic state of the gesture according to the hand posture and the preset static and dynamic state distinguishing method includes the following steps 31-32;

[0146] Step 31, determining the angle range of the roll angle of the back of the hand and obtaining the determination result;

[0147] Step 32, when the roll angle of the back of the hand is within a first preset angle range, determining that the gesture is in a static state;

[0148] When the rolling angle of the back of the hand is within a second preset angle range, it is determined that the gesture is in a dynamic state.

[0149] Further, in an optional implementation of the embodiment of the present invention, generating a control instruction according to the hand posture, the dynamic and static state of the gesture and the preset control instruction generation method includes the following steps 41-42:

[0150] Step 41, when the gesture is in a static state, the flexion and extension angles of two designated fingers are selected as control instruction identifiers, an acceleration control instruction including a corresponding speed is generated according to the flexion and extension angle of one designated finger, and a deceleration control instruction including a corresponding speed is generated according to the flexion and extension angle of the other designated finger, and the corresponding speed is obtained by mapping the corresponding flexion and extension angle through a first linear mapping model;

[0151] In an embodiment of the present invention, any two fingers among the four fingers provided with flexible sensors can be designated, and the flexion and extension angles of the two designated fingers can be used as control instruction identifiers. For example, the flexion and extension angles of the first finger (little finger) and the third finger (middle finger) can be designated as control instruction identifiers.

[0152] In the embodiment of the present invention, since the flexion and extension angle range of the finger is 0-90°, taking the speed interval as 0-Vm / s as an example, the first linear mapping model expression is:

[0153]

[0154] Wherein, v represents the speed value contained in the acceleration control instruction or the deceleration control instruction, V represents the given maximum speed, and α represents the flexion and extension angle of the finger.

[0155] The speed V is determined according to the performance of the actual controlled equipment. For example, if the actual controlled equipment is a drone and the maximum speed of the drone is 10 m / s, the speed V may be 10 m / s.

[0156] Step 42, when the gesture is in a dynamic state, processing the flexion and extension angles to determine whether there is a faulty flexible sensor, and if so, eliminating the flexion and extension angle data collected by the faulty flexible sensor;

[0157] The roll angle of the back of the hand, the pitch angle of the back of the hand, the heading angle of the back of the hand, and the average of the flexion and extension angles of the fingers are selected as control command identifiers, and dynamic control commands are generated according to the average of the roll angle, pitch angle, heading angle and flexion and extension angle. The dynamic control commands include: forward action command, backward action command, ascending action command, descending action command, left turn action command, right turn action command and hovering action command.

[0158] In the embodiment of the present invention, the processing of the flexion and extension angles to determine whether there is a faulty flexible sensor further includes the following steps 421 to 425:

[0159] Step 421, based on the flexion and extension angle mean value calculation model, the flexion and extension angle is calculated and processed to obtain the flexion and extension angle mean value;

[0160] In the embodiment of the present invention, the flexion and extension angle mean value calculation model expression is:

[0161]

[0162] in, represents the mean flexion and extension angle, α i represents the flexion and extension angle of the i-th finger.

[0163] It should be noted that, when i=1, the i-th finger represents the little finger; when i=2, the i-th finger represents the ring finger; when i=3, the i-th finger represents the middle finger; when i=4, the i-th finger represents the index finger.

[0164] Step 422, based on the flexion-extension angle variance calculation model, the flexion-extension angle and the flexion-extension angle mean are calculated to obtain the flexion-extension angle variance;

[0165] In the embodiment of the present invention, the flexion-extension angle variance calculation model expression is:

[0166]

[0167] Among them, σ 2 represents the flexion-extension angle variance.

[0168] Step 423, determining whether the flexion and extension angle variance is less than a variance threshold, and obtaining a determination result;

[0169] In the embodiment of the present invention, the specific value of the variance threshold is set according to the actual situation. For example, the variance threshold is a value within the range of 8 to 10 degrees.

[0170] Step 424, when the judgment result is yes, it is determined that there is no faulty flexible sensor;

[0171] When the judgment result is no, it is determined that there is a faulty flexible sensor, and the step 425 is continued;

[0172] Step 425, based on the flexion-extension angle deviation calculation model, the flexion-extension angle and the flexion-extension angle mean are calculated and processed to obtain the flexion-extension angle deviation, the values ​​of the flexion-extension angle deviation are compared, and the flexible sensor corresponding to the largest flexion-extension angle deviation is determined as a faulty flexible sensor.

[0173] In the embodiment of the present invention, the flexion-extension angle deviation calculation model expression is:

[0174]

[0175] Among them, Δα i represents the flexion and extension angle deviation of the i-th finger.

[0176] In an embodiment of the present invention, taking into account that in actual application, if a flexible sensor among multiple flexible sensors fails, the average of the flexion and extension angles of the finger obtained based on the multiple flexible sensors may have a large error. At the same time, considering that the probability of multiple flexible sensors failing at the same time is extremely small, generally a single flexible sensor fails. Therefore, before generating dynamic control instructions, the flexion and extension angle data is first processed through the above process to determine whether there is a faulty flexible sensor. By eliminating the flexion and extension angle data of the faulty flexible sensor, that is, not using the flexion and extension angle data of the faulty flexible sensor when generating dynamic control instructions, the correctness of the control instructions output in the dynamic state can be ensured.

[0177] Further, in an optional implementation of the embodiment of the present invention, the generating of the dynamic control instruction according to the average of the roll angle, the pitch angle, the heading angle and the flexion and extension angle further comprises the following steps:

[0178] If the rolling angle is within the range of 90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a forward motion command is generated;

[0179] If the rolling angle is within the range of -90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a backward motion command is generated;

[0180] If the pitch angle changes, and the average value of the flexion and extension angle is between 0° and m°, an ascending action command or a descending action command is generated according to the change value of the pitch angle;

[0181] If the heading angle changes and the mean value of the flexion and extension angle is within 90°±m°, a left turn action command or a right turn action command is generated according to the change value of the heading angle;

[0182] If the roll angle is within ±m° and the mean flexion and extension angle is within 90°±m°, a hovering action command is generated;

[0183] Wherein, m represents a preset error parameter.

[0184] In the embodiment of the present invention, the specific value of the error parameter is set according to the actual situation. For example, the error parameter is 5°.

[0185] Specifically, when the gesture recognition device is used for controlling a drone, that is, when the actual controlled equipment is a drone, the above-defined method can be used to generate dynamic control instructions based on the average values ​​of the roll angle, pitch angle, heading angle and flexion and extension angle.

[0186] Further, in the embodiment of the present invention, when the pitch angle of the back of the hand becomes larger, an ascending motion instruction is generated, and when the pitch angle of the back of the hand becomes smaller, a descending motion instruction is generated.

[0187] Furthermore, the ascending action command and the descending action command also include pitch angle information, which is used for controlling the pitch angle of the actual controlled equipment. The pitch angle information is obtained by mapping the pitch angle of the back of the hand, for example, through a one-to-one linear mapping, that is, the pitch angle included in the ascending action command and the descending action command is equal to the pitch angle of the back of the hand.

[0188] Further, in the embodiment of the present invention, when the heading angle of the back of the hand becomes larger, a right turn action instruction is generated, and when the heading angle of the back of the hand becomes smaller, a left turn action instruction is generated.

[0189] Furthermore, the right turn action command and the left turn action command also include heading angle information, which is used for heading angle control of the actual controlled equipment. The heading angle information is obtained by mapping the heading angle of the back of the hand, for example, through a one-to-one linear mapping, that is, the heading angle included in the right turn action command and the left turn action command is equal to the heading angle of the back of the hand.

[0190] It can be seen that the gesture recognition method provided in the embodiment of the present invention is capable of detecting and identifying hand gestures, and distinguishing the dynamic and static states of gestures and generating control instructions based on the hand gesture recognition results. It can solve the problem of gesture recognition stability under arbitrary postures, as well as the problem of distinguishing the dynamic and static states and the problem of generating control instructions, and meet the control requirements of the actual controlled equipment.

[0191] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0192] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0193] Finally, it should be noted that the multi-sensor fusion gesture recognition device and method disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion gesture recognition device, characterized in that: The device comprises: a glove body, an inertial sensor, a flexible sensor, a data processing unit and a user interface; The glove body is used to be worn on the user's hand and to fix the inertial sensor, the flexible sensor, the data processing unit and the user interface; The inertial sensor, the flexible sensor and the user interface are respectively connected to the data processing unit; The inertial sensor is arranged at the back of the glove body, and is used to collect and send quaternions, acceleration and angular velocity to the data processing unit; The flexible sensors are respectively arranged at the little finger, ring finger, middle finger and index finger of the glove body, and are used to collect and send the flexion and extension angles of the fingers to the data processing unit; The data processing unit is used to receive and process the quaternion, the acceleration, the angular velocity and the flexion and extension angle, obtain the hand posture, determine the dynamic and static state of the gesture according to the hand posture and a preset dynamic and static state distinguishing method, and generate a control instruction according to the hand posture, the dynamic and static state of the gesture and a preset control instruction generating method; The user interface can be connected to an external device for data interaction.

2. A gesture recognition method, characterized in that: The method applied to the multi-sensor fusion gesture recognition device according to claim 1 comprises: Acquiring original data, wherein the original data includes: quaternion and flexion and extension angle; Calculating and processing the raw data to obtain the hand posture, wherein the hand posture includes: a roll angle of the back of the hand, a pitch angle of the back of the hand, a heading angle of the back of the hand, and flexion and extension angles of four fingers; Determining the dynamic and static state of the gesture according to the hand posture and a preset dynamic and static state distinguishing method; A control instruction is generated according to the hand posture, the dynamic and static state of the gesture, and a preset control instruction generation method.

3. The gesture recognition method according to claim 2, characterized in that: The obtaining of original data comprises: Initializing the gesture recognition device; Using the inertial sensor, collecting and acquiring the quaternion; The flexion and extension angle is acquired by utilizing the flexible sensor.

4. The gesture recognition method according to claim 2, characterized in that: The calculating and processing the raw data to obtain the hand posture includes: Normalizing the quaternion to obtain a normalized quaternion; Based on the angle solution model, the normalized quaternion is calculated and processed to obtain the roll angle, pitch angle and heading angle of the back of the hand; The flexion and extension angle corresponding to each finger is determined according to the acquired flexion and extension angle.

5. The gesture recognition method according to claim 4, characterized in that: The normalization expression is: in, Represents the i+1th element component of the normalized quaternion, i=0,1,2,3, qi represents the i+1th element component of the quaternion, q0 represents the 1st element component of the quaternion, q1 represents the 2nd element component of the quaternion, q2 represents the 3rd element component of the quaternion, and q3 represents the 4th element component of the quaternion.

6. The gesture recognition method according to claim 5, characterized in that: The angle calculation model expression is: in, represents the roll angle of the back of the hand, arctan represents the calculation of the inverse tangent trigonometric function, θ represents the pitch angle of the back of the hand, arcsin represents the calculation of the inverse sine trigonometric function, and ψ represents the heading angle of the back of the hand.

7. The gesture recognition method according to claim 2, characterized in that: The preset dynamic and static state distinction method includes: When the roll angle of the back of the hand is within a first preset angle range, it is determined that the gesture is in a static state; When the roll angle of the back of the hand is within a second preset angle range, the gesture is determined to be in a dynamic state; The first preset angle range is [120°, 180°), and the second preset angle range is (-180°, 120°).

8. The gesture recognition method according to claim 2, characterized in that: The generating of the control instruction according to the hand posture, the dynamic and static state of the gesture and the preset control instruction generating method includes: When the gesture is in a static state, the flexion and extension angles of two specified fingers are selected as control instruction identifiers, and an acceleration control instruction containing a corresponding speed is generated according to the flexion and extension angle of one of the specified fingers. A deceleration control instruction containing a corresponding speed is generated according to the flexion and extension angle of the other specified finger. The corresponding speed is obtained by mapping the corresponding flexion and extension angle through a first linear mapping model.

9. The gesture recognition method according to claim 8, characterized in that: The generating of the control instruction according to the hand posture, the dynamic and static state of the gesture and the preset control instruction generating method also includes: When the gesture is in a dynamic state, the flexion and extension angle is processed to determine whether there is a faulty flexible sensor, and if so, the flexion and extension angle data collected by the faulty flexible sensor is eliminated; The roll angle of the back of the hand, the pitch angle of the back of the hand, the heading angle of the back of the hand, and the average of the flexion and extension angles of the fingers are selected as control instruction identifiers, and dynamic control instructions are generated according to the average of the roll angle, pitch angle, heading angle and flexion and extension angle. The dynamic control instructions include: forward action instructions, backward action instructions, ascending action instructions, descending action instructions, left turn action instructions, right turn action instructions and hovering action instructions.

10. The gesture recognition method according to claim 9, characterized in that: Generating a dynamic control instruction according to the average of the roll angle, the pitch angle, the heading angle and the flexion and extension angle includes: If the rolling angle is within the range of 90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a forward motion command is generated; If the rolling angle is within the range of -90°±m° and the mean value of the flexion and extension angle is within the range of 90°±m°, a backward motion command is generated; If the pitch angle changes, and the average value of the flexion and extension angle is between 0° and m°, an ascending action command or a descending action command is generated according to the change value of the pitch angle; If the heading angle changes and the mean value of the flexion and extension angle is within 90°±m°, a left turn action command or a right turn action command is generated according to the change value of the heading angle; If the roll angle is within ±m° and the mean flexion and extension angle is within 90°±m°, a hovering action command is generated; Wherein, m represents the preset error parameter.

Images