Gesture recognition method, device and electronic device

Abstract

The present application provides a gesture recognition method, device and electronic device, wherein the gesture recognition method includes: obtaining a hand posture signal reflecting the curvature of the fingers of an operator's hand; obtaining an arm posture signal reflecting the gravitational acceleration of the operator's arm; obtaining a hand posture recognition result corresponding to the hand posture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand posture signal and the hand posture recognition result; obtaining an arm posture recognition result corresponding to the arm posture signal according to a second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result; obtaining a gesture recognition result of the operator according to the hand posture recognition result and the arm posture recognition result; and performing a corresponding operation according to the gesture recognition result.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and more specifically, to a gesture recognition method, device, and electronic device. Background Art

[0002] Gesture recognition is a topic in computer science and language technology that aims to identify human gestures using mathematical algorithms. Gesture recognition can identify movements from various parts of the body, but typically refers to hand movements. Users can use simple gestures to control or interact with devices, allowing computers to understand human behavior.

[0003] In existing technologies, gesture recognition based on optical markers can be affected by factors such as lighting and occlusion, resulting in poor stability. Gesture recognition based on data gloves can only recognize hand gestures, and its scalability is limited. While gesture recognition based on gyroscopes and accelerometers offers greater freedom in gesture measurement and can capture a wider range of gestures, the calculation process is complex and prone to sensor drift, leading to errors in gesture recognition results. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a gesture recognition method, device, and electronic device that can solve the problems of low gesture scalability and poor stability of gesture recognition results.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a gesture recognition method, comprising:

[0007] Acquiring a hand posture signal reflecting the curvature of fingers of the operator's hand;

[0008] Acquiring an arm posture signal reflecting the gravity acceleration of the operator's arm;

[0009] Obtaining a hand gesture recognition result corresponding to the hand gesture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result;

[0010] Obtaining an arm posture recognition result corresponding to the arm posture signal according to a second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result;

[0011] Obtaining a gesture recognition result of the operator according to the hand gesture recognition result and the arm gesture recognition result;

[0012] Execute corresponding operations according to the gesture recognition result.

[0013] Optionally, obtaining a hand posture signal reflecting the curvature of fingers of the operator's hand includes:

[0014] Obtaining a hand electrical signal that satisfies a first stability condition; wherein the hand electrical signal includes multiple electrical signals that respectively reflect the curvature of each finger of the operator, and the first stability condition includes a deviation between a currently sampled hand electrical signal and hand electrical signals obtained by N previous samplings being within a first set range, where N is greater than or equal to 1;

[0015] The hand gesture signal is obtained according to the hand electrical signal that meets the first stability condition.

[0016] Optionally, obtaining the hand gesture signal according to the hand electrical signal that satisfies the first stability condition includes:

[0017] Normalization processing is performed on the hand electrical signal that meets the first stability condition to obtain the hand posture signal.

[0018] Optionally, obtaining an arm posture signal reflecting the gravitational acceleration of the operator's arm includes:

[0019] Acquire a gravity acceleration signal reflecting the operator's arm posture; wherein the gravity acceleration signal includes a first gravity acceleration signal reflecting the upper arm posture and a second gravity acceleration signal reflecting the operator's lower arm posture;

[0020] The arm posture signal is obtained according to the first gravity acceleration signal and the second gravity acceleration signal.

[0021] Optionally, the gravity acceleration signal satisfies a second stability condition; the second stability condition includes: the total deviation between the gravity acceleration signal of the current acquisition period and the gravity acceleration signal of the previous M sampling periods is within a second set range, and M is greater than or equal to 1.

[0022] Optionally, the gravity acceleration signal includes gravity acceleration values ​​of the x-axis, y-axis and z-axis; M=4;

[0023] The total deviation is expressed as:

[0024]

[0025] Where i=0 represents the first gravity acceleration signal, i=1 represents the second gravity acceleration signal, a=x represents the gravity acceleration value of the x-axis, a=y represents the gravity acceleration value of the y-axis, a=z represents the gravity acceleration value of the z-axis, t represents the sampling period, T is the current sampling period, T-4, T-3, T-2, T-1, T represent the first four consecutive sampling periods, D 2 Represents the square of the derivative of the acceleration due to gravity.

[0026] Optionally, the step of constructing the first model includes:

[0027] The obtained hand gesture signal sample sets corresponding to the preset hand gestures;

[0028] Constructing training data, verification data, and test data based on the hand gesture signal sample sets corresponding to the preset hand gestures;

[0029] Obtain at least two first basic models corresponding to the first model;

[0030] Using the training data and the verification data to train the model parameters of each of the first basic models, respectively, to obtain a second basic model corresponding to each of the first basic models;

[0031] Using the test data to test the preset performance indicators of each second basic model, and obtaining an indicator value of the preset performance indicator corresponding to each second basic model;

[0032] The second basic model whose index value meets the preset conditions is used as the first model.

[0033] Optionally, the second model is a data comparison table reflecting the second mapping relationship.

[0034] In a second aspect, an embodiment of the present application provides a gesture recognition device, comprising:

[0035] A first acquisition module is used to acquire a hand posture signal reflecting the curvature of the fingers of the operator's hand;

[0036] A second acquisition module is used to acquire an arm posture signal reflecting the gravity acceleration of the operator's arm;

[0037] a hand gesture recognition module, configured to obtain a hand gesture recognition result corresponding to the hand gesture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result;

[0038] an arm posture recognition module, configured to obtain an arm posture recognition result corresponding to the arm posture signal according to a second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result;

[0039] a gesture recognition module, configured to obtain a gesture recognition result of the operator based on the hand gesture recognition result and the arm gesture recognition result;

[0040] The execution module is used to perform corresponding operations according to the gesture recognition result.

[0041] In a third aspect, an embodiment of the present application provides an electronic device comprising the gesture recognition device as described in the second aspect; or, the electronic device comprises a memory and a processor, the memory being used to store a computer program, and the processor being used to control the electronic device to execute the method steps described in any one of the first aspects under the control of the computer program.

[0042] The gesture recognition method of the embodiment of the present application uses an arm posture signal that reflects the gravitational acceleration of the operator's arm to identify the arm posture. Since the gravitational acceleration is only related to the position and direction of the sensor relative to the ground, the process of obtaining the arm posture signal does not involve integral calculations and data drift problems. The obtained arm posture signal is highly accurate, thereby improving the accuracy of the arm posture recognition results and gesture recognition results. At the same time, the final gesture recognition result is obtained by fusing the arm posture recognition results and the hand posture recognition results, which increases the types of recognizable gestures and improves the scalability of gestures. Furthermore, since the acquisition process of the hand posture signal and the arm posture signal is not affected by the external environment, the stability of the gesture recognition results can be improved.

[0043] Other features and advantages of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the specification.

[0045] Figure 1 is a structural diagram of a gesture recognition system that can be used to implement the gesture recognition method according to an embodiment of the present application;

[0046] Figure 2 is a flowchart of a gesture recognition method according to one embodiment;

[0047] Figure 3 is a schematic diagram of correctly assembling an inertial sensor according to one embodiment;

[0048] Figure 4 is a schematic diagram of a format for recording hand electrical signal samples according to one embodiment;

[0049] Figure 5 is a schematic diagram of display effects of samples in each sample set in three-dimensional space according to one embodiment;

[0050] Figure 6 is a schematic diagram of test results of the accuracy of prediction results of various models according to one embodiment;

[0051] Figure 7 is a flowchart of an example of a gesture recognition method according to an embodiment;

[0052] Figure 8 is a principle block diagram of a gesture recognition device according to one embodiment;

[0053] Figure 9 is a schematic structural diagram of an electronic device according to an embodiment;

[0054] Figure 10 is a schematic structural diagram of an electronic device according to another embodiment. DETAILED DESCRIPTION

[0055] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0056] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0057] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0058] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0059] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0060] <Hardware Configuration>

[0061] Figure 1Schematic diagram of the structure of a gesture recognition system that can be used to implement the gesture recognition method of the embodiment of the present application.

[0062] like Figure 1 As shown, the gesture recognition system 1000 includes a data glove 1100, a first inertial sensor 1200, a second inertial sensor 1300, a core processor 1400, and a smart terminal 1500. The data glove 1100, the first inertial sensor 1200, and the second inertial sensor 1300 are respectively connected to the core processor 1400 for communication, and the core processor 1400 is connected to the smart terminal 1500 for communication.

[0063] In one embodiment, the data glove 1100 includes a glove body, five flexible piezoresistive sensors, a wiring module and a first data acquisition module.

[0064] In this embodiment, the glove body has an interlayer and resembles a standard glove. Five flexible piezoresistive sensors and a wiring harness are located within the interlayer of the glove body. Each of the five flexible piezoresistive sensors is a thin, slender strip, located in the interlayer on the backs of the five fingers of the glove body, from the joints connecting the fingers to the palm to the fingertips. These strips cover the back of each finger and deform with finger movement. The wiring harness, located within the interlayer on the back of the glove body, connects the five flexible piezoresistive sensors to the first data acquisition module, streamlining the wiring between the sensors and the first data acquisition module.

[0065] In this embodiment, the first data acquisition module is disposed on the back of the glove body near the wrist. The first data acquisition module includes a voltage acquisition chip, a first processor, and a first wireless communication module.

[0066] In this embodiment, the voltage acquisition chip is used to collect multi-channel voltage signals output by five flexible piezoresistive sensors, and send the collected multi-channel voltage signals to the first processor through a serial peripheral interface (SPI).

[0067] In this embodiment, upon receiving multiple voltage signals, the first processor performs smoothing and average filtering on the multiple voltage signals; calculates multiple resistance signals reflecting the curvature of each finger of the operator based on the processed multiple voltage signals; generates a hand electrical signal based on the calculated multiple resistance signals; and transmits the hand electrical signal to the first wireless communication module via a serial peripheral interface (SPI). The first processor may be an ARM processor or other processors, without specific limitation herein.

[0068] In this embodiment, upon receiving the hand electrical signal, the first wireless communication module sends the hand electrical signal to the core processor 1400. The first wireless communication module may be a WIFI module or other module capable of wireless communication, which is not specifically limited herein.

[0069] The surface of the flexible piezoresistive sensor is a layer of special resistive material. When the flexible piezoresistive sensor is subjected to stress and bends, its surface resistance changes. The greater the bend, the greater the resistance. This means that there is a corresponding relationship between the degree of finger bending and the resistance of the flexible piezoresistive sensor. Therefore, the resistance of the flexible piezoresistive sensor can reflect the degree of finger bending.

[0070] In one embodiment, the first inertial sensor 1200 and the second inertial sensor 1300 have the same structure. The hardware structures of the first inertial sensor 1200 and the second inertial sensor 1300 are described below using the first inertial sensor as an example.

[0071] In this embodiment, the first inertial sensor 1200 includes an elastic band and a second data acquisition module. The second data acquisition module is arranged on the elastic band and can be fixed to a corresponding position of the operator's arm by elastic force.

[0072] In this embodiment, the second data acquisition module includes a posture sensor, a second processor and a second wireless communication module.

[0073] In this embodiment, the attitude sensor can collect gravitational acceleration signals along the x-axis, y-axis, and z-axis, and transmit the collected three-axis gravitational acceleration signals to the second processor. The attitude sensor can be a MEMS accelerometer or other motion sensor capable of obtaining three-axis gravitational acceleration signals, and is not specifically limited herein.

[0074] In this embodiment, upon receiving a gravity acceleration signal, the second processor performs smoothing and mean filtering on the gravity acceleration signal and transmits the processed gravity acceleration signal to the second wireless communication module via a serial peripheral interface (SPI). The second processor may be an ARM processor or other processors, without specific limitation.

[0075] In this embodiment, upon receiving the processed gravity acceleration signal, the second wireless communication module sends the processed gravity acceleration signal to the core processor 1400. The second wireless communication module may be a WIFI module or other module capable of wireless communication, which is not specifically limited herein.

[0076] When the arm posture changes, the gravity acceleration signals of the posture sensor in the x-axis, y-axis, and z-axis directions will change. That is, there is a corresponding relationship between the arm posture and the three-axis gravity acceleration signals. Therefore, the operator's arm posture can be reflected by the three-axis gravity acceleration signals.

[0077] In one embodiment, the operator's arm posture includes an upper arm posture and a lower arm posture. To separately obtain the upper arm posture and lower arm posture of the operator, a first inertial sensor can be fixed to the middle of the operator's upper arm to obtain a first gravity acceleration signal reflecting the upper arm posture. A second inertial sensor can be fixed to a position on the operator's lower arm near the wrist to obtain a second gravity acceleration signal reflecting the operator's lower arm posture. The first gravity acceleration signal and the second gravity acceleration signal constitute a gravity acceleration signal reflecting the operator's arm posture.

[0078] In one embodiment, the core processor 1400 includes a third wireless communication module, a third processor, a smart chip, and a data link module.

[0079] In this embodiment, when the third wireless communication module receives the hand electrical signal and the gravity acceleration signal, it sends the hand electrical signal and the gravity acceleration signal to the third processor; the third wireless communication module can be a WIFI module or other module that can realize wireless communication, which is not specifically limited here.

[0080] In this embodiment, when the hand electrical signal and gravity acceleration signal are determined to be stable according to a preset stability condition, the third processor obtains a hand gesture signal based on the hand electrical signal that satisfies the first stability condition; obtains an arm gesture signal based on the gravity acceleration signal that satisfies the stability condition; transmits the hand gesture signal and arm gesture signal to the smart chip; and generates corresponding operation instructions based on the gesture recognition results returned by the smart chip and transmits them to the data link module. The third processor can be an ARM processor or other processor, which is not specifically limited here.

[0081] In this embodiment, the smart chip stores a preset first model, a second model, and a third model. When a hand gesture signal is received, a hand gesture recognition result corresponding to the hand gesture signal can be obtained based on the first model. When an arm gesture signal is received, an arm gesture recognition result corresponding to the arm gesture signal can be obtained based on the second model. When a hand gesture recognition result and an arm gesture recognition result are obtained, a gesture recognition result corresponding to the hand gesture recognition result and the arm gesture recognition result can be obtained based on the third model, and the obtained gesture recognition result can be sent to a third processor.

[0082] After receiving the operation instruction corresponding to the gesture recognition result, the data link module sends the operation instruction to the smart terminal 1500, and the smart terminal responds to the operation instruction corresponding to the operation instruction.

[0083] It should be understood that although Figure 1 Only one data glove 1100, a first inertial sensor 1200, a second inertial sensor 1300, a core processor 1400 and a smart terminal 1500 is shown, but this does not mean that their respective numbers are limited. The system can include multiple data gloves 1100, first inertial sensors 1200, second inertial sensors 1300, core processors 1400 and smart terminals 1500.

[0084] Hereinafter, various embodiments and examples according to the present invention will be described with reference to the accompanying drawings.

[0085] <Method Example>

[0086] Figure 2 is a flowchart of a gesture recognition method according to an embodiment.

[0087] like Figure 2 As shown, the gesture recognition method of this embodiment may include steps S210-S260.

[0088] S210 , obtaining a hand gesture signal reflecting the curvature of fingers of the operator's hand.

[0089] When the operator makes different hand gestures, the degree of curvature of each finger will change. For example, when the operator makes the hand gesture of "2", the corresponding curvature of each finger can be: the curvature of the extended index finger and middle finger is 0 degrees, and the curvature of the thumb, ring finger and little finger is 180 degrees. For another example, when the operator makes the hand gesture of "0", the corresponding curvature of each finger can be 180 degrees. For another example, when the operator makes the hand gesture of "5", the corresponding curvature of each finger can be 0 degrees. It can be seen that the curvature of each finger has a certain correspondence with the hand gesture. Therefore, hand gesture recognition can be performed through the hand gesture signal reflecting the curvature of the fingers of the operator's hand.

[0090] In one embodiment, obtaining a hand gesture signal reflecting the curvature of fingers of the operator's hand includes steps S211 and S212.

[0091] S211 , obtaining a hand electrical signal that meets a first stability condition; wherein the hand electrical signal includes multiple electrical signals that respectively reflect the bending degree of each finger of the operator.

[0092] In one embodiment, the Figure 1The data glove 1100 in the data glove 1100 collects the operator's hand electrical signals. When the operator wears the data glove 1100, he can make hand gestures to be identified. The five flexible piezoresistive sensors in the data glove 1100 output voltage signals according to the curvature of the corresponding fingers. When the first data acquisition module in the data glove 1100 receives the five voltage signals output by the five flexible piezoresistive sensors, it pre-processes the five voltage signals respectively, and calculates five resistance signals reflecting the curvature of each finger based on the pre-processed five voltage signals; a hand electrical signal is generated based on the calculated five resistance signals, that is, the hand electrical signal includes multiple electrical signals reflecting the curvature of each finger of the operator, and the multiple electrical signals are multiple resistance signals.

[0093] In one embodiment, the data glove 1100 collects hand electrical signals at a preset first frequency and transmits the collected hand electrical signals to the core processor 1400 at a preset second frequency. The first and second frequencies can be pre-set frequencies or configured based on specific application scenarios, and are not specifically limited here. The first and second frequencies can be the same or different, and are not specifically limited here.

[0094] For example, the first frequency of the data glove 1100 is pre-set to be equal to the second frequency of 100 Hz. At this time, the data glove 1100 will collect hand electrical signals at a sampling frequency of 100 times per second, and send the collected hand electrical signals to the core processor 1400 in real time.

[0095] The operator's hand gesture is a process from movement to stillness. When the hand gesture remains unchanged within the set time, the hand electrical signal output by the data glove 1100 is also in a stable state. The hand gesture at this time is the hand gesture to be recognized.

[0096] In one embodiment, in order to improve the accuracy of hand gesture recognition, after receiving the hand electrical signal sent by the data glove 1100, the core processor 1400 first determines whether the current hand electrical signal is stable based on a preset first stability condition. If it is determined that the current hand electrical signal is stable, the hand electrical signal that meets the first stability condition is obtained to execute step S212.

[0097] In one embodiment, the first stability condition may include a deviation between a currently sampled hand electrical signal and N previously sampled hand electrical signals being within a first set range, where N is greater than or equal to 1. N and the first set range may be pre-set values ​​or configured based on specific application scenarios, and are not specifically limited herein.

[0098] For example, N is pre-set to 19, the first setting range is 0, and the first stability condition is that when the hand electrical signal obtained by the current sampling is the same as the hand electrical signal obtained by the previous 19 samplings, it is considered that the hand electrical signal obtained by the current sampling meets the first stability condition.

[0099] S212: Obtain a hand gesture signal according to the hand electrical signal that meets the first stability condition.

[0100] In one embodiment, obtaining the hand gesture signal according to the hand electrical signal that meets the first stability condition includes: performing normalization processing on the hand electrical signal that meets the first stability condition to obtain the hand gesture signal.

[0101] S220: Acquire an arm posture signal reflecting the gravity acceleration of the operator's arm.

[0102] When the operator adopts different arm postures, the position and orientation of any arm position relative to the ground changes, causing the gravity acceleration signal collected by the inertial sensor fixed to the operator's arm position to change. Therefore, the gravity acceleration value corresponding to the arm position has a certain correspondence with the arm posture. Therefore, arm posture recognition can be performed using the arm posture signal reflecting the gravity acceleration of the operator's arm.

[0103] In one embodiment, obtaining an arm posture signal reflecting the gravity acceleration of the operator's arm includes steps S221 and S222.

[0104] S221, obtaining a gravity acceleration signal reflecting the operator's arm posture; wherein the gravity acceleration signal includes a first gravity acceleration signal reflecting the upper arm posture and a second gravity acceleration signal reflecting the operator's forearm posture.

[0105] The arm posture consists of the upper arm posture and the forearm posture. For example, the arm posture can be horizontal to the ground with the forearm turned outward; the arm posture can be vertical to the ground with the upper arm pointing downward; the upper arm can also be horizontal to the ground with the forearm vertical to the ground, etc.

[0106] To accurately identify the operator's arm posture, a first inertial sensor is fixed in the middle of the operator's upper arm to obtain a first gravity acceleration signal reflecting the upper arm posture. A second inertial sensor is fixed near the operator's forearm near the wrist to obtain a second gravity acceleration signal reflecting the operator's forearm posture.

[0107] In one embodiment, the first inertial sensor and the second inertial sensor can both obtain three-axis gravity acceleration signals of the x-axis, y-axis and z-axis. When the operator wears the first inertial sensor and the second inertial sensor correctly, the matching diagram of the x-axis, y-axis and z-axis of the two inertial sensors and the arm direction is as shown in FIG. Figure 3 As shown, when the operator's arm hangs naturally, the five fingers are stretched out, and the palm is facing backward, the x-axes of the two inertial sensors point to the fingertips, and the y-axis and z-axis satisfy the right-hand coordinate system with the x-axis.

[0108] The arm posture made by the operator is also a process from dynamic to static. When the arm posture remains unchanged within the set time, the gravity acceleration signal will also be in a stable state. The arm posture at this time is the arm posture to be identified.

[0109] In one embodiment, in order to improve the accuracy of arm posture recognition, after the current sampling period ends, the core processor 1400 first determines whether the gravity acceleration signal meets the preset second stability condition. If the second stability condition is met, the gravity acceleration signal last obtained in the current sampling period is used as the gravity acceleration signal that meets the second stability condition to execute step S222.

[0110] In one embodiment, the second stability condition includes: a total deviation between the gravity acceleration signal of the current acquisition period and the gravity acceleration signals of the previous M sampling periods is within a second set range, and M is greater than or equal to 1. M and the second set range may be pre-set values ​​or configured based on specific application scenarios, and are not specifically limited herein.

[0111] In one embodiment, the gravity acceleration signal includes gravity acceleration signals of the x-axis, y-axis and z-axis, and the corresponding gravity acceleration signal includes a first gravity acceleration signal and a second gravity acceleration signal which also include gravity acceleration signals of the x-axis, y-axis and z-axis.

[0112] In this embodiment, the deviation of the gravity acceleration signal in the current sampling period is equal to the sum of the squares of the derivatives of the three-axis gravity acceleration signals in the first gravity acceleration signal and the three-axis gravity acceleration signals in the second gravity acceleration signal, a total of 6-axis gravity acceleration signals with respect to time.

[0113] When M=4 is preset and the second setting range is 0, the total deviation can be expressed as:

[0114]

[0115] Where i=0 represents the first gravity acceleration signal, i=1 represents the second gravity acceleration signal, a=x represents the gravity acceleration value of the x-axis, a=y represents the gravity acceleration value of the y-axis, a=z represents the gravity acceleration value of the z-axis, t represents the sampling period, T is the current sampling period, T-4, T-3, T-2, T-1, T represent the first four consecutive sampling periods, D 2 Represents the square of the derivative of gravitational acceleration with respect to time.

[0116] When the operator's arm posture is stable, the time derivative of the gravity acceleration signal for each axis should be zero. However, due to the unavoidable high-frequency vibrations of the human body, which are detected by the inertial sensors, the time derivatives of the gravity acceleration signals for each axis tend to fluctuate around zero. To mitigate the effects of high-frequency vibrations on stability assessment, a preset derivative threshold is typically used. When the time derivative of the gravity acceleration signal for any axis is less than or equal to the preset derivative threshold, the time derivative of the gravity acceleration signal for that axis is set to zero. Based on multiple experimental analyses, this derivative threshold can be set to 0.3, or it can be adjusted based on the specific application scenario. This is not a specific setting here.

[0117] S222: Obtain an arm posture signal according to the first gravity acceleration signal and the second gravity acceleration signal.

[0118] The gravity acceleration signal ranges from -9.8 to 9.8. When the positive direction of an axis points perpendicular to the ground, the acceleration value of the corresponding axis is +9.8, and vice versa.

[0119] In an embodiment where the first gravity acceleration signal and the second gravity acceleration signal both include gravity acceleration signals of the x-axis, the y-axis, and the z-axis, in order to improve the efficiency of arm posture recognition, step S222 includes steps S222-1 to S222-3.

[0120] S222-1, mapping the three-axis gravity acceleration signal values ​​of the first gravity acceleration signal into three reference quantities of 0, 1, and 2 respectively through a preset data mapping step to obtain a mapped first gravity acceleration signal.

[0121] The following steps describe the data mapping process, using the x-axis as an example: First, divide the x-axis gravity acceleration signal value by 9.8 to obtain the first x-axis mapping data. According to the data mapping table shown in Table 1, the first x-axis mapping data is mapped to one of three values: 0, 1, or 2, to obtain the second x-axis mapping data. Using the x-axis as an example, the second x-axis mapping data is obtained for the y-axis and the z-axis. Based on the second x-axis, y-axis, and z-axis mapping data, the mapped first gravity acceleration signal is obtained.

[0122] For example, when the second mapping data of the x-axis is 1, the second mapping data of the y-axis is 2, and the second mapping data of the z-axis is 1, the mapped first gravity acceleration signal is [1, 2, 1].

[0123] Table 1:

[0124] Mapping rules Reference quantity First mapping data>first mapping threshold 2 Second mapping threshold < first mapping data < first mapping threshold 1 First mapping data<second mapping threshold 0

[0125] The second mapping threshold in Table 1 is smaller than the first mapping threshold. For example, the first mapping threshold may be 0.6, and the second mapping threshold may be -0.6.

[0126] S222-2, referring to step S222-1, the three-axis gravity acceleration signal values ​​of the second gravity acceleration signal are mapped into three reference quantities of 0, 1, and 2 respectively to obtain the mapped second gravity acceleration signal.

[0127] S222-3: Obtain an arm posture signal according to the mapped first gravity acceleration signal and the mapped second gravity acceleration signal.

[0128] In one embodiment, the arm posture signal may be obtained according to the second mapping data of the x-axis in the first gravity acceleration signal and the second gravity acceleration signal.

[0129] For example, the first gravity acceleration signal is [1, 1, 2], and the second mapping data of the x-axis in the second gravity acceleration signal is 0, then the arm posture signal is [1, 1, 2, 0].

[0130] S230: Obtain a hand gesture recognition result corresponding to the hand gesture signal according to the first model, wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result.

[0131] In one embodiment, the step of constructing the first model includes steps S231 to S236.

[0132] S231: Obtain a hand gesture signal sample set corresponding to each preset hand gesture.

[0133] In this embodiment, step S231 specifically includes steps S231 - 1 to S231 - 2 .

[0134] S231-1, obtaining a hand electrical signal sample set corresponding to each preset hand posture.

[0135] In this embodiment, the data glove 1100 collects hand electrical signal samples corresponding to each preset hand posture, and the hand electrical signals are automatically recorded using a python script. The process is as follows:

[0136] 1) When the script starts, enter the number of the hand gesture to be recorded.

[0137] The numbering is generally numerical and increments sequentially. For example, the first hand gesture to be recorded may be numbered 1, the second hand gesture to be recorded may be numbered 2, the third hand gesture to be recorded may be numbered 3, and so on. The numbers of the hand gestures to be recorded are set accordingly.

[0138] 2) After entering the hand gesture number to be recorded, a 5-second countdown begins. Make the desired hand gesture while keeping your fingers as still as possible. When the countdown ends, the script records the hand signal sample corresponding to the current hand gesture and displays the sample and the number of recorded hand signal samples on the terminal device.

[0139] 3) After quickly moving the finger, return to the hand posture to be recorded and maintain it for 0.2 seconds. At this time, the script will record the hand electrical signal that is stable for 0.2 seconds as a valid hand electrical signal sample and update the number of recorded hand electrical signal samples.

[0140] After the displayed hand electrical signal sample is recorded, if the finger remains motionless, the same hand electrical signal sample will not be recorded repeatedly. Therefore, it is necessary to quickly move the finger and then return to the hand posture to be recorded to collect the hand electrical signal sample.

[0141] 4) Repeat step 3) until the number of recorded hand electrical signal samples is within a first range, and then terminate the Python terminal. The first range may be 100-200.

[0142] After the Python terminal is interrupted, the collection of hand electrical signal samples is stopped. The script writes the recorded hand electrical signal samples into a file in the format of: / data / [i] / YYYY_MM_DD_hh_mm_ss.txt. The format of the hand electrical signal samples recorded in each txt file is as follows: Figure 4 As shown in the figure, there are n rows and 5 columns. n is the number of valid hand electrical signal samples corresponding to the hand posture to be recorded. 5 means that each hand electrical signal contains 5 channels, one for each finger. The value under each channel is the resistance signal value reflecting the curvature of the finger.

[0143] 5) All recorded hand electrical signal samples are used as a hand electrical signal sample set corresponding to the hand posture to be recorded.

[0144] 6) Repeat steps 1) to 5) to obtain a hand electrical signal sample set corresponding to each preset hand posture.

[0145] S231 - 2 , pre-processing the hand electrical signal sample sets corresponding to the preset hand postures to obtain hand posture signal sample sets corresponding to the preset hand postures.

[0146] In one embodiment, the preprocessing mainly includes:

[0147] 1) Clear out the outlier samples in each sample set.

[0148] In the process of collecting hand electrical signal samples corresponding to each preset hand posture, some intermediate states of the fingers will be regarded as valid movements when they are deformed. At this time, the hand electrical signal samples collected by the data glove 1100 are significantly different from the hand electrical signal samples corresponding to the hand posture that needs to be collected, and should be eliminated to improve the accuracy of the collected hand electrical signal samples.

[0149] 2) Perform linear normalization on the samples in each sample set.

[0150] The value range of each resistance signal in the hand electrical signal output by the data glove 1100 is between [50, 250]. Considering the small amount of data, in order to improve the training convergence speed of the model, the linear normalization method is used to scale the value range of each resistance signal in the hand electrical signal to between [0, 1].

[0151] 3) Use principal component analysis (PCA) to reduce the sample data in each sample set after linear normalization to 3 dimensions, and display all 3D sample data in 3D space. Different colors or shapes can be used to mark samples in different sample sets, so that it is possible to clearly determine whether there are obvious boundaries between samples in different sample sets in 3D space. If there are obvious boundaries between samples in different sample sets, specifically Figure 5 As shown, the normalized sample set corresponding to each preset hand posture is used as the hand posture signal sample set corresponding to each preset hand posture. If there is no obvious boundary between samples in different sample sets, the data preprocessing step needs to be re-executed.

[0152] S232: Construct training data, verification data, and test data based on the hand gesture signal sample sets corresponding to the preset hand gestures.

[0153] In this embodiment, the training data and the validation data are used to train and optimize the parameters of the first basic model to obtain the second basic model. The test data is used to test the preset performance indicators of the second basic model to obtain the index values ​​of the preset performance indicators of the second basic model.

[0154] In one embodiment, 20% of the samples are extracted from each set of hand gesture signal samples corresponding to each preset hand gesture to construct test data. From the remaining 80% of the samples in each sample set, 20% of the samples are extracted to construct the first subset. The process of constructing the first subset is repeated until five subsets are obtained. A 5-fold cross-validation is employed, with each subset used as validation data and the remaining four subsets used as training data. In this way, five sets of training data and validation data are generated. The parameters of the first basic model are trained and optimized based on each of the five sets of training data and validation data.

[0155] S233: Obtain at least two first basic models corresponding to the first model.

[0156] In this embodiment, the first basic model is a machine learning model that can be used for classification, for example, a logistic regression model, a naive Bayes model, a support vector machine model (SVM), a decision tree model, or a random forest model.

[0157] S234: Use the training data and the verification data to train the model parameters of each first basic model respectively to obtain a second basic model corresponding to each first basic model.

[0158] S235 , using the test data to test the preset performance indicators of each second basic model, and obtaining the indicator value of the preset performance indicator corresponding to each second basic model.

[0159] In this embodiment, the preset performance indicator may be the mean value of the accuracy of the prediction results, the convergence speed of the model training, or other indicator values ​​reflecting the performance of the model, which are not specifically limited here.

[0160] S236: The second basic model whose index value meets the preset conditions is used as the first model.

[0161] In one embodiment, the preset performance indicator includes a mean value of the accuracy of the prediction results.

[0162] In this embodiment, the second basic model with the largest mean value of prediction result accuracy may be selected as the first model.

[0163] In one embodiment, a logistic regression model, a naive Bayes model, a support vector machine model SVM, a decision tree model, and a random forest model RandomForest are selected, totaling five first basic models.

[0164] In this embodiment, the training data and validation data constructed in step S232 are used to train and optimize the model parameters of the five first basic models to obtain five second basic models. The test data constructed in step S232 are used to test the five second basic models respectively. The test results of the accuracy of the prediction results of each model are as follows: Figure 6 shown. Figure 6In the figure, the dots represent the mean of the prediction accuracy, and the I-shaped marks represent the variance of the prediction accuracy. It can be seen that the second basic model based on Random Forest has the largest mean of the preset result accuracy, so the second basic model based on Random Forest is selected as the final first model.

[0165] After training is completed, the final first model is saved to models / [model name] / YYYY_MM_DD_hh_mm_ss.pkl using Python's json.dump method.

[0166] S240: Obtain an arm posture recognition result corresponding to the arm posture signal according to the second model, wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result.

[0167] In one embodiment, the second model is a data comparison table that reflects the second mapping relationship. The data comparison table may be as shown in Table 2.

[0168] Table 2:

[0169] X1 Y1 Z1 X2 gesture 1 1 2 1 Forearms and upper arms are level with the ground, palms facing down 1 0 1 1 Forearms and upper arms are level with the ground, forearms turned outwards 1 2 1 1 Forearms and upper arms are level with the ground, forearms turned inward 1 1 0 1 Forearms and upper arms are level with the ground, palms facing up 2 1 1 2 Forearm and upper arm vertically upwards from the ground 0 1 1 0 Forearm and upper arm vertically downward 1 1 0 0 Upper arms vertical to the ground, lower arms horizontal to the ground, palms facing up 1 1 2 0 Upper arms vertical to the ground, lower arms horizontal to the ground, palms facing downwards 1 0 1 0 The upper arm is vertical to the ground, the forearm is horizontal to the ground, and the forearm is turned outward 2 1 1 1 The upper arm is horizontal to the ground, and the lower arm is vertical to the ground and upward

[0170] In Table 2, the arm inertial navigation status is relatively rare, so only the x-axis data is used for classification. X1, Y1, and Z1 represent the gravity acceleration signal values ​​of the arm inertial navigation sensor on the x, y, and z axes, and X2 represents the gravity acceleration signal value of the arm inertial sensor on the x axis.

[0171] S250 , obtaining a gesture recognition result of the operator according to the hand gesture recognition result and the arm gesture recognition result.

[0172] Specifically for gesture recognition, the recognition of hand gestures and arm gestures are independent of each other, that is, a person can make hand gestures and arm gestures at the same time. finger There are N arm postures im Theoretically, the combination of the two can have N=N finger ×N imu By fusing the arm gesture recognition results with the hand gesture recognition results to obtain the final gesture recognition result, the types of recognizable gestures can be increased, improving the scalability of gestures.

[0173] In one embodiment, the recognition result of the hand posture is five fingers open, and the recognition result of the arm posture is the upper arm vertical to the ground, the forearm horizontal to the ground, and the palm facing down. Then the gesture recognition result of the operator is the upper arm vertical to the ground, the forearm horizontal to the ground, five fingers open, and the palm facing down.

[0174] S260: Execute corresponding operations according to the gesture recognition result.

[0175] In this embodiment, a control instruction may be generated according to the gesture recognition result, and the control instruction is used to control the smart terminal 1500 to perform a corresponding operation.

[0176] In one embodiment, the intelligent terminal 1500 may be a drone. When the core processor 1400 obtains the gesture recognition result, it may generate a control instruction for controlling the drone according to the gesture recognition result, and control the drone to perform a corresponding operation through the control instruction.

[0177] For example, when the gesture recognition result is: the upper arm is vertical to the ground, the forearm is horizontal to the ground, the five fingers are open, and the palm is facing down, the core processor 1400 generates an instruction to control the landing of the drone based on the gesture recognition result. After receiving the landing instruction, the drone performs the landing operation.

[0178] The above steps S210 and S220 can be performed simultaneously or in a preset order, which is not specifically limited here. The above steps S230 and S240 can be performed simultaneously or in a preset order, which is not specifically limited here.

[0179] The gesture recognition method of the embodiment of the present application uses an arm posture signal that reflects the gravitational acceleration of the operator's arm to identify the arm posture. Since the gravitational acceleration is only related to the position and direction of the sensor relative to the ground, the process of obtaining the arm posture signal does not involve integral calculations and data drift problems. The accuracy of the obtained arm posture signal is high, thereby improving the accuracy of the arm posture recognition results and the gesture recognition results. At the same time, the final gesture recognition result is obtained by fusing the arm posture recognition results and the hand posture recognition results, which increases the types of recognizable gestures and improves the scalability of gestures. Furthermore, since the acquisition process of the hand posture signal and the arm posture signal is not affected by the external environment, the stability of the gesture recognition results can be improved.

[0180] <Example>

[0181] Figure 7 is a flowchart of an example of a gesture recognition method according to an embodiment.

[0182] like Figure 7 As shown, the method may include steps S701 to S000.

[0183] In this example, the gesture made by the operator is a gesture for controlling the landing of the drone.

[0184] S701: Acquire a hand electrical signal that meets a first stability condition.

[0185] The hand electrical signal includes a multi-channel resistance signal that reflects the bending degree of each finger of the operator. The first stability condition includes that the hand electrical signal obtained by current sampling is the same as the hand electrical signal obtained by previous 19 samplings.

[0186] S702: Perform normalization processing on the hand electrical signal that meets the first stability condition to obtain a hand posture signal.

[0187] S703: Obtain a hand gesture recognition result corresponding to the hand gesture signal according to the first model.

[0188] The first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result, wherein the hand gesture recognition result is five fingers open.

[0189] S704: Acquire a gravity acceleration signal that meets a second stability condition.

[0190] In this embodiment, the gravity acceleration signal includes a first gravity acceleration signal reflecting the posture of the operator's upper arm and a second gravity acceleration signal reflecting the posture of the operator's lower arm.

[0191] In this embodiment, the second stability condition includes: the total deviation between the gravity acceleration signal in the current sampling period and the gravity acceleration signals in the previous four sampling periods is zero. The deviation of the gravity acceleration signal in the current sampling period is equal to the sum of the squares of the derivatives of the three-axis gravity acceleration signals in the first gravity acceleration signal and the three-axis gravity acceleration signals in the second gravity acceleration signal with respect to time, for a total of six axes. The total deviation can be expressed as:

[0192]

[0193] Where i=0 represents the first gravity acceleration signal, i=1 represents the second gravity acceleration signal, a=x represents the gravity acceleration value of the x-axis, a=y represents the gravity acceleration value of the y-axis, a=z represents the gravity acceleration value of the z-axis, t represents the sampling period, T is the current sampling period, T-4, T-3, T-2, T-1, T represent the first four consecutive sampling periods, D 2 Represents the square of the derivative of gravitational acceleration with respect to time.

[0194] S705 : Obtain an arm posture signal according to the first gravity acceleration signal and the second gravity acceleration signal.

[0195] This step specifically includes: mapping the three-axis gravity acceleration signal values ​​of the first gravity acceleration signal into three reference quantities of 0, 1, and 2 through a preset data mapping step to obtain a mapped first gravity acceleration signal. Referring to the data mapping step of the first gravity acceleration signal, mapping the three-axis gravity acceleration signal values ​​of the second gravity acceleration signal into three reference quantities of 0, 1, and 2 to obtain a mapped second gravity acceleration signal. Based on the mapped first gravity acceleration signal and the mapped second gravity acceleration signal, an arm posture signal is obtained. The arm posture signal is: [1, 1, 2, 0].

[0196] S706: Obtain an arm posture recognition result corresponding to the arm posture signal according to the data comparison table reflecting the second mapping relationship, wherein the second mapping relationship is a mapping relationship between the arm posture signal and the arm posture recognition result.

[0197] In this embodiment, the data comparison table reflecting the second mapping relationship is shown in Table 2. By querying the data comparison table, the arm posture recognition result corresponding to the arm posture signal [1, 1, 2, 0] is obtained as the upper arm is vertical to the ground, the lower arm is horizontal to the ground, and the palm is downward.

[0198] S707 , obtaining a gesture recognition result of the operator according to the hand gesture recognition result and the arm gesture recognition result.

[0199] In this embodiment, when the hand posture recognition result is five fingers open and the arm posture recognition result is the upper arm vertical to the ground, the forearm horizontal to the ground, and the palm facing down, the operator's gesture recognition result is: the upper arm vertical to the ground, the forearm horizontal to the ground, the five fingers open, and the palm facing down.

[0200] S708: Control the drone to perform corresponding operations based on the gesture recognition result.

[0201] In this embodiment, based on the gesture recognition result of the upper arm being vertical to the ground, the lower arm being horizontal to the ground, and the five fingers being open with the palm facing downward, a command for controlling the drone to land is generated. The drone responds to the command and executes the landing operation.

[0202] The gesture recognition method of the embodiment of the present application uses an arm posture signal that reflects the gravitational acceleration of the operator's arm to identify the arm posture. Since the gravitational acceleration is only related to the position and direction of the sensor relative to the ground, the process of obtaining the arm posture signal does not involve integral calculations and data drift problems. The accuracy of the obtained arm posture signal is high, thereby improving the accuracy of the arm posture recognition results and the gesture recognition results. At the same time, the final gesture recognition result is obtained by fusing the arm posture recognition results and the hand posture recognition results, which increases the types of recognizable gestures and improves the scalability of gestures. Furthermore, since the acquisition process of the hand posture signal and the arm posture signal is not affected by the external environment, the stability of the gesture recognition results can be improved.

[0203] <Device Example>

[0204] Figure 8 is a principle block diagram of a gesture recognition device according to an embodiment.

[0205] like Figure 8 As shown, the gesture recognition device 8000 may include:

[0206] The first acquisition module 8100 is used to acquire a hand gesture signal reflecting the curvature of the fingers of the operator's hand;

[0207] The second acquisition module 8200 is used to acquire an arm posture signal reflecting the gravity acceleration of the operator's arm;

[0208] A hand gesture recognition module 8300 is configured to obtain a hand gesture recognition result corresponding to the hand gesture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result;

[0209] The arm posture recognition module 8400 is configured to obtain an arm posture recognition result corresponding to the arm posture signal according to the second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result;

[0210] The gesture recognition module 8500 is used to obtain the operator's gesture recognition result based on the hand gesture recognition result and the arm gesture recognition result;

[0211] The execution module 8600 is used to perform corresponding operations according to the gesture recognition results.

[0212] In one embodiment, the first acquisition module 8100 is specifically used to obtain a hand electrical signal that meets a first stability condition; wherein the hand electrical signal includes multiple electrical signals that respectively reflect the curvature of each finger of the operator, and the first stability condition includes that the deviation between the hand electrical signal obtained by the current sampling and the hand electrical signal obtained by the previous N samplings is within a first set range, and N is greater than or equal to 1; a hand posture signal is obtained based on the hand electrical signal that meets the first stability condition.

[0213] In one embodiment, when obtaining the hand gesture signal based on the hand electrical signal that meets the first stability condition, the first acquisition module 8100 is specifically configured to normalize the hand electrical signal that meets the first stability condition to obtain the hand gesture signal.

[0214] In one embodiment, the second acquisition module 8200 is specifically used to obtain a gravity acceleration signal reflecting the operator's arm posture; wherein the gravity acceleration signal includes a first gravity acceleration signal reflecting the upper arm posture and a second gravity acceleration signal reflecting the operator's forearm posture; the arm posture signal is obtained based on the first gravity acceleration signal and the second gravity acceleration signal.

[0215] In one embodiment, the gravity acceleration signal satisfies a second stability condition; the second stability condition includes: the total deviation between the gravity acceleration signal of the current acquisition period and the gravity acceleration signal of the previous M sampling periods is within a second set range, and M is greater than or equal to 1.

[0216] In one embodiment, the gravity acceleration signal includes gravity acceleration values ​​of the x-axis, y-axis, and z-axis; M=4; and the total deviation is expressed as:

[0217]

[0218] Where i=0 represents the first gravity acceleration signal, i=1 represents the second gravity acceleration signal, a=x represents the gravity acceleration value of the x-axis, a=y represents the gravity acceleration value of the y-axis, a=z represents the gravity acceleration value of the z-axis, t represents the sampling period, T is the current sampling period, T-4, T-3, T-2, T-1, T represent the first four consecutive sampling periods, D 2 Represents the square of the derivative of the acceleration due to gravity.

[0219] In one embodiment, the hand gesture recognition module 8300 also includes a first model construction module, which is specifically used to obtain a hand gesture signal sample set corresponding to each preset hand gesture; construct training data, verification data and test data based on the hand gesture signal sample set corresponding to each preset hand gesture; obtain at least two first basic models corresponding to the first model; use the training data and verification data to train the model parameters of each first basic model respectively, and obtain the second basic model corresponding to each first basic model; use the test data to test the preset performance indicators of each second basic model, and obtain the index value of the preset performance indicator corresponding to each second basic model; use the second basic model whose index value meets the preset conditions as the first model.

[0220] In one embodiment, the second model is a data comparison table reflecting the second mapping relationship.

[0221] <Electronic Equipment Example>

[0222] Figure 9 is a schematic structural diagram of an electronic device according to an embodiment.

[0223] like Figure 9 As shown, the electronic device 9000 includes a memory 9200 and a processor 9100. The memory 9200 is used to store computer programs, and the processor 9100 is used to control the electronic device 9000 to execute the method described in any method embodiment of this specification under the control of the computer program.

[0224] Each module of the electronic device 9000 can be implemented by the processor 9100 in this embodiment executing a computer program stored in the memory 9200, or can be implemented by other circuit structures, which is not limited here.

[0225] Figure 10 is a schematic structural diagram of an electronic device according to another embodiment.

[0226] like Figure 10 As shown, the electronic device A000 includes the above gesture recognition device 8000.

[0227] One or more embodiments of the present specification may be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present specification.

[0228] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0229] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0230] The computer program instructions for performing the operation of the embodiments of this specification can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, programming languages ​​include object-oriented programming languages-such as Smalltalk, C++ etc., and conventional procedural programming languages-such as " C " language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (such as utilizing an Internet service provider to connect by the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing the various aspects of this specification.

[0231] Various aspects of this specification are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0232] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0233] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0234] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and a part of the module, program segment or instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0235] The embodiments of this specification have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A gesture recognition method, characterized in that: include: Acquiring a hand posture signal reflecting the curvature of fingers of the operator's hand; Acquiring an arm posture signal reflecting the gravity acceleration of the operator's arm; Obtaining a hand gesture recognition result corresponding to the hand gesture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result; Obtaining an arm posture recognition result corresponding to the arm posture signal according to a second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result; Obtaining a gesture recognition result of the operator according to the hand gesture recognition result and the arm gesture recognition result; According to the gesture recognition result, the corresponding operation is performed. The step of obtaining an arm posture signal reflecting the gravitational acceleration of the operator's arm includes: Acquire gravitational acceleration signals along the x-axis, y-axis, and z-axis that reflect the operator's arm posture; wherein the gravitational acceleration signals include a first gravitational acceleration signal that reflects the upper arm posture and a second gravitational acceleration signal that reflects the operator's lower arm posture; The arm posture signal is obtained according to the first gravity acceleration signal and the second gravity acceleration signal.

2. The method according to claim 1, characterized in that The acquiring of the hand gesture signal reflecting the curvature of the fingers of the operator's hand comprises: Obtaining a hand electrical signal that satisfies a first stability condition; wherein the hand electrical signal includes multiple electrical signals that respectively reflect the curvature of each finger of the operator, and the first stability condition includes a deviation between a currently sampled hand electrical signal and hand electrical signals obtained by N previous samplings being within a first set range, where N is greater than or equal to 1; The hand gesture signal is obtained according to the hand electrical signal that meets the first stability condition.

3. The method according to claim 2, characterized in that Obtaining the hand gesture signal according to the hand electrical signal that meets the first stability condition includes: Normalization processing is performed on the hand electrical signal that meets the first stability condition to obtain the hand posture signal.

4. The method according to claim 1, wherein The gravity acceleration signal satisfies a second stability condition; the second stability condition includes: a total deviation between the gravity acceleration signal of the current acquisition period and the gravity acceleration signal of the previous M sampling periods is within a second set range, and M is greater than or equal to 1.

5. The method according to claim 4, characterized in that The gravity acceleration signal includes the gravity acceleration values ​​of the x-axis, y-axis and z-axis; M=4; The total deviation is expressed as: Where i=0 represents the first gravity acceleration signal, i=1 represents the second gravity acceleration signal, a=x represents the gravity acceleration value of the x-axis, a=y represents the gravity acceleration value of the y-axis, a=z represents the gravity acceleration value of the z-axis, t represents the sampling period, T is the current sampling period, T-4, T-3, T-2, T-1, T represent the first four consecutive sampling periods, D 2 Represents the square of the derivative of the acceleration due to gravity.

6. The method according to claim 1, characterized in that The steps of constructing the first model include: The obtained hand gesture signal sample sets corresponding to the preset hand gestures; Constructing training data, verification data, and test data based on the hand gesture signal sample sets corresponding to the preset hand gestures; Obtain at least two first basic models corresponding to the first model; Using the training data and the verification data to train the model parameters of each of the first basic models, respectively, to obtain a second basic model corresponding to each of the first basic models; Using the test data to test the preset performance indicators of each second basic model, and obtaining an indicator value of the preset performance indicator corresponding to each second basic model; The second basic model whose index value meets the preset conditions is used as the first model.

7. The method according to claim 1, characterized in that The second model is a data comparison table reflecting the second mapping relationship.

8. A gesture recognition device, characterized in that: include: A first acquisition module is used to acquire a hand posture signal reflecting the curvature of the fingers of the operator's hand; A second acquisition module is used to acquire an arm posture signal reflecting the gravity acceleration of the operator's arm; a hand gesture recognition module, configured to obtain a hand gesture recognition result corresponding to the hand gesture signal according to a first model; wherein the first model reflects a first mapping relationship between the hand gesture signal and the hand gesture recognition result; an arm posture recognition module, configured to obtain an arm posture recognition result corresponding to the arm posture signal according to a second model; wherein the second model reflects a second mapping relationship between the arm posture signal and the arm posture recognition result; a gesture recognition module, configured to obtain a gesture recognition result of the operator based on the hand gesture recognition result and the arm gesture recognition result; An execution module, configured to execute corresponding operations according to the gesture recognition result, The second acquisition module is specifically configured to: Acquire gravitational acceleration signals along the x-axis, y-axis, and z-axis that reflect the operator's arm posture; wherein the gravitational acceleration signals include a first gravitational acceleration signal that reflects the upper arm posture and a second gravitational acceleration signal that reflects the operator's lower arm posture; The arm posture signal is obtained according to the first gravity acceleration signal and the second gravity acceleration signal.

9. An electronic device comprising the gesture recognition device according to claim 8; or The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to control the electronic device to execute the method steps according to any one of claims 1 to 7 under the control of the computer program.

Images