Gesture segmentation and recognition method, electronic equipment and storage medium

Abstract

The invention discloses a gesture segmentation and recognition method, electronic equipment and a storage medium, and belongs to the technical field of gesture recognition. In order to achieve rapid and accurate gesture recognition, the method comprises the steps that gesture action data are collected through an intelligent data glove, the collected gesture action data are divided into n pieces of continuous window data through a fixed sliding window, each piece of window data is divided into three sub-windows with the same size, and the sub-windows are divided into n pieces of continuous window data; using a distance calculation function to calculate internal similar features between the sub-windows of each window, establishing an adjacent window similar feature judgment method for judging whether the gesture window is a basic gesture action window or a transition gesture action window, and respectively obtaining the basic gesture action window and the transition gesture action window; and performing up-sampling processing on the transition gesture action window, and respectively inputting the obtained basic gesture action window and the processed transition gesture action window into a gesture action recognition model for recognition to obtain a basic gesture action and a transition gesture action.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture recognition, and in particular relates to a gesture segmentation and recognition method, an electronic device and a storage medium. Background Art

[0002] With the rapid development of wearable technology and artificial intelligence technology, gesture recognition technology has been widely used in various fields. Gesture recognition technology based on multi-sensor data gloves has also been widely used in the field of human-computer interaction due to its advantages such as unlimited usage scenarios and obvious interactive features. Gesture segmentation is a pre-technique for gesture recognition. Its segmentation effect directly determines the accuracy of gesture recognition. The existing fixed threshold segmentation algorithm has extremely poor segmentation effect for new users with different operating habits, and it is difficult to achieve personalized adaptation for different users.

[0003] The invention patent with application number 202311327282.1 and invention name “A high-precision gesture data recognition method, electronic device and storage medium” discloses the technical solution of “collecting new target gesture data, constructing a new target gesture data set, obtaining source domain gesture data, and constructing a source domain gesture data set; constructing a useless gesture filtering model based on the mPUL algorithm and the TSC algorithm; filtering the new target gesture data using the constructed useless gesture filtering model to obtain target domain gesture data and construct a target domain gesture data set; constructing a cross-domain gesture recognition model based on transfer learning; inputting the collected source domain gesture data set and target domain gesture data set into the constructed cross-domain gesture recognition model based on transfer learning, performing gesture recognition from the source domain to the target domain, and obtaining high-precision gesture data recognition results.” However, the above technology does not consider the detection and recognition of transition gesture actions, and uses a deep learning algorithm to filter useless gestures, which requires high computing resources and has a slow inference speed. Summary of the invention

[0004] The problem to be solved by the present invention is to quickly and accurately recognize gestures, and a gesture segmentation and recognition method, an electronic device and a storage medium are proposed.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A gesture segmentation and recognition method comprises the following steps:

[0007] S1. Use the smart data glove to collect gesture data, and use a fixed sliding window to divide the collected gesture data into n continuous window data to obtain a continuous window data set X = {x 1 ,x 2 ,…,x n}, where the i-th window data is marked as x i ;

[0008] S2. Divide each window data obtained in step S1 into three sub-windows of equal size. Then the i-th window data is divided into three sub-window data, namely

[0009] S3. For the sub-window data of each window obtained in step S2, use the distance calculation function to calculate the internal similarity features between the sub-windows of each window;

[0010] S4. Based on the internal similarity features between the sub-windows of each window obtained in step S4, the absolute difference between the minimum value of the internal similarity features in the previous window and the maximum value of the internal similarity features in the current window, and the absolute difference between the minimum value of the internal similarity features in the current window and the maximum value of the internal similarity features in the previous window are calculated, and a method for determining similarity features of adjacent windows is established to determine whether the gesture window is a basic gesture action window or a transition gesture action window, and obtain a basic gesture action window and a transition gesture action window respectively;

[0011] S5. Upsampling the transition gesture action window obtained in step S4;

[0012] S6. Input the obtained basic gesture action window and the processed transition gesture action window into the gesture action recognition model for recognition, respectively, to obtain the basic gesture action and the transition gesture action.

[0013] Furthermore, in step S1, the sensors on the smart data glove are a three-axis accelerometer and a three-axis gyroscope, and the i-th window data x i ∈R w×C , w is the window size, and C is the number of channels of the sensor.

[0014] Furthermore, the specific implementation method of step S3 includes the following steps:

[0015] S3.1. For the i-th window x i , the internal similarity feature of the i-th window The calculation formula is as follows:

[0016]

[0017] in, is the internal similarity feature of the first sub-window of the i-th window, is the internal similarity feature of the second sub-window of the i-th window, is the internal similarity feature of the third sub-window of the i-th window, and d is the Manhattan distance calculation function; the Manhattan distance calculation function is as follows

[0018] d(sx p ,sxq )=∑|sx p -sx q |

[0019] Among them, d(sx p ,sx q ) refers to the distance between sub-window p and sub-window q;

[0020] S3.2. Use the accelerometer and gyroscope data to calculate the Manhattan distance function, that is, replace sx in the formula p ,sx q Use the accelerometer and gyroscope data to replace and obtain the internal similarity features f for the p-th sub-window and the q-th sub-window pq The calculation formula is:

[0021]

[0022] Among them, a is the acceleration data, g is the gyroscope data, and the window is divided into three sub-windows. Represents the average acceleration value of the j-axis in the q-th sub-window, It represents the average angular velocity value of the j-axis in the q-th sub-window, the j-axis is the x-coordinate axis, the y-coordinate axis or the z-coordinate axis, p=1 or 2 or 3, q=1 or 2 or 3, and p and q are not equal.

[0023] Furthermore, the specific implementation method of step S4 includes the following steps:

[0024] S4.1. Set the i-1th window as the previous window, the i-th window as the current window, and calculate the absolute value b between the minimum value of the internal similarity feature in the previous window and the maximum value of the internal similarity feature in the current window i-1 , the calculation formula is:

[0025]

[0026] Among them, min is the minimum function and max is the maximum function;

[0027] S4.2. Calculate the absolute difference e between the minimum value of the similarity feature within the current window and the maximum value of the similarity feature within the previous window i , the calculation formula is:

[0028]

[0029] S4.3. Establish a method for determining similar features of adjacent windows:

[0030] S4.3.1. Set the detection flag representing the transition gesture action window to Tag t , the default is Tagt Set to non-transition gesture, the transition gesture action start threshold is TH ts , the transition gesture end threshold is TH te ;

[0031] S4.3.2. In Tag t For non-transition gestures, e i The transition gesture action starts at the threshold TH ts For comparison, if e i Greater than TH ts , the previous window is regarded as the basic gesture action window, and the current window is regarded as the beginning of the transition gesture action window, so the current window is added to the transition gesture action window X tans ; Otherwise, the previous window is considered the transition gesture action window.

[0032] Furthermore, the specific implementation method of step S5 includes the following steps:

[0033] S5.1. In Tag t In the case of a transition gesture action window, by adding the current window to the buffer X tans To start the aggregation transition window; the aggregation process is to convert b i-1 The transition gesture ends at the threshold TH te For comparison, if b i-1 Above the threshold TH te , it is considered that the end of the transition gesture action window will be reached, and the previous window will be aggregated as the end of the basic gesture action;

[0034] All detected transition windows are aggregated into X tans , then, for X tans Upsampling is performed to resize it to a fixed window size to match the input size of the recognition model; otherwise, window aggregation continues in the next round;

[0035] S5.2. When the process of aggregating the windows exceeds the maximum length of the transition gesture action, it is processed as a static gesture in the basic gesture action;

[0036] S5.3. When both windows have mixed samples of transition gestures and basic gestures, the values ​​e of the two adjacent windows are higher than the threshold transition gesture start threshold TH. ts In this case, the latest window is considered to be the beginning of the transition gesture action window.

[0037] Furthermore, the specific implementation method of step S6 includes the following steps:

[0038] S6.1. Construct two gesture recognition models, each of which is a CNN-LSTM network model architecture;

[0039] S6.2. The obtained basic gesture action window and the processed transition gesture action window are respectively input into the two gesture action recognition models constructed in step S6.1 for recognition to obtain the basic gesture action and the transition gesture action.

[0040] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a gesture segmentation and recognition method when executing the computer program.

[0041] A computer-readable storage medium stores a computer program, which implements the gesture segmentation and recognition method when executed by a processor.

[0042] Beneficial effects of the present invention:

[0043] The gesture segmentation and recognition method described in the present invention achieves accurate segmentation of continuous gestures by calculating similar features within a window and dissimilar features of adjacent windows, and based on the difference in motion patterns between transition gestures and basic gestures, and can clearly distinguish between basic gestures and transition gestures in continuous gestures. In addition, the gesture segmentation and recognition method based on a fixed sliding window is improved, and a large window is divided into smaller windows to reduce confusion of multiple gesture information caused by excessively large windows. The segmented incomplete data is aggregated to ensure the integrity of the gesture data in the data window, thereby significantly improving the accuracy of continuous gesture segmentation, with a gesture segmentation accuracy rate of up to 94.8%.

[0044] Compared with the traditional gesture segmentation method, the gesture segmentation and recognition method described in the present invention is usually included in the segmented gesture data window in the transition gesture, resulting in a decrease in gesture recognition effect. However, the present invention fully considers the influence of transition gestures in the continuous gesture segmentation process, and avoids the interference of transition gestures (invalid information) on the basic gesture recognition performance by separating the basic gesture action from the transition gesture action, ensuring that only valid information is included in the basic gesture data window, thereby significantly improving the recognition accuracy of the basic gesture. In addition, the present invention adopts two independent gesture recognition models to independently train and recognize the segmented transition gestures and basic gestures, so that each model focuses on its own gesture action set, thereby improving the recognition accuracy. Experimental results show that the recognition accuracy of basic gestures reaches 96.3%, and the recognition rate of transition gestures reaches 93.2%.

[0045] The gesture segmentation and recognition method described in the present invention has the advantages of low computing resource requirements and high efficiency, and supports a minimum gesture interval time of 0.2 seconds in continuous gesture segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a gesture segmentation and recognition method according to the present invention;

[0047] Figure 2 A framework diagram of a gesture segmentation and recognition method according to the present invention;

[0048] Figure 3 It is a differential pattern diagram of the transition gesture and the basic gesture signal of the present invention;

[0049] Figure 4 : These are dissimilar feature graphs between two situations of two consecutive windows of the present invention, wherein (a) is a situation where a basic gesture action before a transition gesture action results in a lower feature value in the current window, while the feature value in the previous window is medium-low or high, and (b) is a situation where a transition gesture action before a basic gesture action results in a medium-low or high feature value in the current window, while the feature value in the previous window is low. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0051] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0052] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 4 The detailed instructions are as follows:

[0053] Embodiment 1:

[0054] A gesture segmentation and recognition method comprises the following steps:

[0055] S1. Use the smart data glove to collect gesture data, and use a fixed sliding window to divide the collected gesture data into n continuous window data to obtain a continuous window data set X = {x 1 ,x 2 ,…,x n}, where the i-th window data is marked as x i ;

[0056] Furthermore, in step S1, the sensors on the smart data glove are a three-axis accelerometer and a three-axis gyroscope, and the i-th window data x i ∈R w×C , w is the window size, C is the number of channels of the sensor;

[0057] S2. Divide each window data obtained in step S1 into three sub-windows of equal size. Then the i-th window data is divided into three sub-window data, namely

[0058] S3. For the sub-window data of each window obtained in step S2, use the distance calculation function to calculate the internal similarity features between the sub-windows of each window;

[0059] Furthermore, the specific implementation method of step S3 includes the following steps:

[0060] S3.1. For the i-th window x i , the internal similarity feature of the i-th window The calculation formula is as follows:

[0061]

[0062] in, is the internal similarity feature of the first sub-window of the i-th window, is the internal similarity feature of the second sub-window of the i-th window, is the internal similarity feature of the third sub-window of the i-th window, and d is the Manhattan distance calculation function; the Manhattan distance calculation function is as follows

[0063] d(sx p ,sx q )=∑|sx p -sx q |

[0064] Among them, d(sx p ,sx q ) refers to the distance between sub-window p and sub-window q;

[0065] S3.2. Use the accelerometer and gyroscope data to calculate the Manhattan distance function, that is, replace sx in the formula p ,sx q Use the accelerometer and gyroscope data to replace and obtain the internal similarity features f for the p-th sub-window and the q-th sub-window pq The calculation formula is:

[0066]

[0067] Among them, a is the acceleration data, g is the gyroscope data, and the window is divided into three sub-windows. Represents the average acceleration value of the j-axis in the q-th sub-window, It represents the average angular velocity value of the j-axis in the q-th sub-window, the j-axis is the x-coordinate axis, the y-coordinate axis or the z-coordinate axis, p=1 or 2 or 3, q=1 or 2 or 3, and p and q are not equal.

[0068] Furthermore, d(sx m ,sx n ) is a distance function for calculating internal similarity. Manhattan distance is used as the distance calculation function between two vectors. The accumulated result of Manhattan distance can reflect the total acceleration and gyroscope changes. For example, a large change in Manhattan distance may indicate fast movement or sudden stop, while a small change may indicate a smoother motion state. This is very useful for distinguishing different gesture action modes. Its definition is as follows:

[0069] d(sx m ,sx n )=∑|sx m -sx n |

[0070] Among them, d(sx m ,sx n ) refers to two subwindows sx m 、sx n The distance between two vectors. Manhattan distance is the distance between two vectors.

[0071] Furthermore, the value of the internal similarity feature is affected by the gesture rate within the window. The more intense the gesture within the window, the greater the difference between the sub-windows. On the contrary, the gentler the hand within the window, the smaller the difference between the sub-windows and the smaller the feature value. Since the basic gesture follows the stimulus in the continuous gesture, the feature value of the basic gesture is higher than that of the transition gesture.

[0072] S4. Based on the internal similarity features between the sub-windows of each window obtained in step S4, the absolute difference between the minimum value of the internal similarity features in the previous window and the maximum value of the internal similarity features in the current window, and the absolute difference between the minimum value of the internal similarity features in the current window and the maximum value of the internal similarity features in the previous window are calculated, and a method for determining similarity features of adjacent windows is established to determine whether the gesture window is a basic gesture action window or a transition gesture action window, and obtain a basic gesture action window and a transition gesture action window respectively;

[0073] Furthermore, the signal characteristics of transition gestures and basic gestures are different (regardless of whether the activity is static or dynamic). The signals of static gestures are usually low frequency and low amplitude, while the signals of dynamic activities are composed of high frequency and high amplitude. On the other hand, the signal frequency of transition gestures is lower, but relatively shorter than that of basic gestures. Therefore, the characteristics of transition gestures and basic gestures are different. A high difference between the features of two consecutive windows indicates that there is a transition gesture window at the beginning or end of the window. Therefore, the segmentation method proposed in the present invention distinguishes transition gestures from basic gestures by calculating two dissimilarity values ​​between two adjacent windows, that is, calculating the degree of dissimilarity between the internal similarity features of two consecutive windows.

[0074] Furthermore, the specific implementation method of step S4 includes the following steps:

[0075] S4.1. Set the i-1th window as the previous window, the i-th window as the current window, and calculate the absolute value b between the minimum value of the internal similarity feature in the previous window and the maximum value of the internal similarity feature in the current window i-1 , the calculation formula is:

[0076]

[0077] Among them, min is the minimum function and max is the maximum function;

[0078] Furthermore, b aims to determine whether the current window is the beginning of a transition gesture action. This value is measured by calculating the absolute difference between the minimum value of the internal similarity feature in the previous window and the maximum value of the internal similarity feature in the current window. If the current window is the beginning of a transition gesture action and the previous window is the end of a basic gesture action, this value is lower;

[0079] S4.2. Calculate the absolute difference e between the minimum value of the similarity feature within the current window and the maximum value of the similarity feature within the previous window i , the calculation formula is:

[0080]

[0081] Furthermore, e is intended to determine whether the current window is the end of a transition gesture. This value is measured by calculating the absolute difference between the minimum value of the similarity feature within the current window and the maximum value of the similarity feature within the previous window. The ratio of this difference is calculated based on the lowest feature value of the current window. Given an internal similarity feature vector fi- 1 ,f i And for the previous window and the current window respectively;

[0082] Further, such as Figure 4 As shown in (a), the current window (x i ) is a transition gesture, and the previous window (x i-1 ) is a basic gesture. In the present invention, since the starting point of each gesture is similar, the amplitude of the transition gesture is relatively small. and The maximum and minimum eigenvalues ​​of the current window indicated are low values ​​because the distance between the subwindows is small and the patterns of the signal data are similar. For the previous window, the maximum eigenvalue is a high value, the minimum eigenvalue is a low or medium value because the basic action roommate is in a stable state or a violent fluctuation state, the signal data fluctuates greatly, and the distance between sub-windows is large. The minimum and maximum eigenvalues ​​are used to determine b i-1 and e i Therefore, in this case, b i-1 and e i The values ​​will be small and large respectively.

[0083] On the contrary, if Figure 4 As shown in (b), the current window (x i ) is the basic gesture action, and the previous window (x i-1 ) is a transition gesture action, The maximum eigenvalue of the current window indicated is high. and the minimum eigenvalue are low or medium; while the previous window and are low values, then by calculation, b i-1 and e i The values ​​will be large and small respectively.

[0084] Therefore, it can be concluded that in all cases, a large value of b can determine the beginning of the basic gesture action, while a large value of e can be used to determine the end of the basic gesture action. The large value, medium value, and small value used here are only used to describe the characteristic values ​​b and e.

[0085] S4.3. Establish a method for determining similar features of adjacent windows:

[0086] S4.3.1. Set the detection flag representing the transition gesture action window to Tag t , the default is Tag t Set to non-transition gesture, the transition gesture action start threshold is TH ts , the transition gesture end threshold is TH te ;

[0087] S4.3.2. In Tag t For non-transition gestures, e i The transition gesture action starts at the threshold TH ts For comparison, if e i Greater than TH ts , the previous window is regarded as the basic gesture action window, and the current window is regarded as the beginning of the transition gesture action window, so the current window is added to the transition gesture action window X tans ; Otherwise, the previous window is considered the transition gesture action window.

[0088] S5. Upsampling the transition gesture action window obtained in step S4;

[0089] Furthermore, the specific implementation method of step S5 includes the following steps:

[0090] S5.1. In Tag t In the case of a transition gesture action window, by adding the current window to the buffer X tans To start the aggregation transition window; the aggregation process is to convert b i-1 The transition gesture ends at the threshold TH te For comparison, if b i-1 Above the threshold TH te , it is considered that the end of the transition gesture action window will be reached, and the previous window will be aggregated as the end of the basic gesture action;

[0091] All detected transition windows are aggregated into X tans , then, for X tans Upsampling is performed to resize it to a fixed window size to match the input size of the recognition model; otherwise, window aggregation continues in the next round;

[0092] S5.2. When the process of aggregating the windows exceeds the maximum length of the transition gesture action, it is processed as a static gesture in the basic gesture action;

[0093] S5.3. When both windows have mixed samples of transition gestures and basic gestures, the values ​​e of two adjacent windows will be higher than the threshold transition gesture start threshold TH. ts , then the latest window is considered to be the beginning of the transition gesture action window.

[0094] S6. The obtained basic gesture action window and the processed transition gesture action window are respectively input into the gesture action recognition model for recognition to obtain the basic gesture action and the transition gesture action;

[0095] Furthermore, the specific implementation method of step S6 includes the following steps:

[0096] S6.1. Construct two gesture recognition models, each of which is a CNN-LSTM network model architecture;

[0097] Furthermore, the CNN-LSTM network model architecture used is to extract the spatial features of multi-dimensional sensor data through two two-dimensional convolutional layers, and then connect the long short-term memory network layer (LSTM) to model the temporal changes. The specific model architecture is that the data is first input into a two-dimensional convolutional layer (Conv2D), and random inactivation (dropout) is used to prevent overfitting of network training, and then through a two-dimensional convolutional layer (Conv2D), and through average pooling (AveragePooling), batch normalization (BatchNormalization), random inactivation (dropout) and flattening (Flatten), to extract and process the data features and perform dimensionality reduction. The data is then input into the long short-term memory network layer (LSTM) to model the temporal changes of the data, and the learned features are fused through the fully connected layer and finally used for classification tasks. The specific configuration parameters are shown in Table 1;

[0098] Table 1 CNN-LSTM gesture recognition model parameters

[0099]

[0100] S6.2. The obtained basic gesture action window and the processed transition gesture action window are respectively input into the two gesture action recognition models constructed in step S6.1 for recognition to obtain the basic gesture action and the transition gesture action.

[0101] Furthermore, two gesture recognition models based on CNN-LSTM networks with the same architecture are used to train and recognize transition gestures and basic gestures, and the recognition task is divided into two models so that each model can focus on learning its own gesture set. After completing the gesture segmentation based on similarity difference, the transition gesture action window and the basic gesture action window will be input into the above model for recognition. The window size is initially set to 60 data (1.2 seconds), and will be adjusted with different gesture recognition requirements in the later stage to adapt to different gesture recognition task requirements.

[0102] In this embodiment, first, the data signal is divided into multiple continuous windows, and then these windows are passed to the continuous gesture segmentation model based on data similarity, and two types of data features (window internal similarity features and adjacent window dissimilarity features) are extracted simultaneously to distinguish between basic gesture action windows and transition gesture action windows. Internal similarity, this is done by dividing the window into three sub-windows of equal size and then measuring the similarity of the signals between these sub-windows. Adjacent window dissimilarity, by measuring the difference between the features of the current window and the features of the previous window in a fixed sliding window, in order to achieve gesture segmentation of the gesture, the current window is the last window read from the sensor. As Figure 2 As shown, mark the current window as x i , the window before the current window is marked with x i-1 . If the difference between two adjacent windows is small, it means that there is no transition gesture action window in any of the two windows, but if the difference is high, it means that there is a transition gesture action window between the two windows. According to the difference, the transition gesture action window and the basic gesture action window can be distinguished. Then, the data window is interpolated and scaled, and the overlong window is downsampled until the window is reduced to the length required by the gesture recognition model; the overshort window is upsampled until the window is expanded to the length required by the gesture recognition model. Finally, the distinguished and processed gesture action window is sent to the gesture action recognizer, where the basic gesture action window is sent to the basic gesture action recognition model, and the transition gesture action window is sent to the transition gesture action recognition model for gesture action recognition. Each gesture recognition model is implemented using LSTM-CNN deep learning.

[0103] Based on the data glove, this paper establishes a continuous gesture dataset to verify the model effect. The dataset contains 1500 continuous gesture sequences, each sequence consists of 10-15 gestures, and the starting and ending points of each basic gesture and transition gesture are manually marked. In order to verify the effectiveness of transition gesture segmentation, two advanced gesture segmentation algorithms are selected for comparison: a gesture segmentation model based on a fixed sliding window and an adaptive differential threshold gesture segmentation model.

[0104] The accuracy of gesture segmentation refers to the overlap rate between each manually segmented gesture (true label) and the gesture automatically segmented by the algorithm. The specific results are shown in Table 2:

[0105] Table 2 Comparison of segmentation accuracy of different gesture segmentation models

[0106]

[0107]

[0108] The experimental results show that the accuracy of the gesture segmentation model based on similarity difference proposed in the present invention is as high as 94.8%, which is better than the other two segmentation models. Although the segmentation accuracy of the adaptive differential threshold gesture segmentation model is very close to that of the model of the present invention (only 0.3% lower), it does not consider transition gestures during segmentation, resulting in the segmented gestures containing both transition gestures and basic gestures, affecting the subsequent recognition effect. To verify this, we input the segmented gestures into a unified recognition model for verification. The recognition models all adopt the CNN-LSTM structure proposed in the present invention to reduce the influence of other variables. The experimental results are shown in Table 3.

[0109] In addition, since the gesture segmentation model based on similarity difference of the present invention has low computing resource requirements and high efficiency, it supports a minimum gesture interval time of 0.2 seconds in continuous gesture segmentation, and has the shortest segmentation time compared with other gesture segmentation models.

[0110] Table 3 Comparison of recognition accuracy of gesture action windows segmented by different gesture segmentation models

[0111]

[0112]

[0113] It can be seen from Table 3 that when the gesture segmentation model based on the fixed sliding window is used, the gesture recognition effect is the worst, with an accuracy of 88.6%. This is because the gesture segments segmented by the model may have missing or redundant data, resulting in poor recognition effect. Secondly, when using the adaptive differential threshold gesture segmentation model, the segmented gesture segments contain useless transition gesture data, which affects the overall recognition effect, and the recognition accuracy is 92.5%. In contrast, the gesture segmentation method based on similarity difference proposed in the present invention considers the segmentation and recognition of transition gestures, and the gesture recognition effect is significantly better than other segmentation algorithms, with a recognition accuracy of 96.3%, and the recognition accuracy of transition gestures reaches 93.2%.

[0114] Embodiment 2:

[0115] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a gesture segmentation and recognition method described in Example 1 when executing the computer program.

[0116] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Moreover, the processor is used to implement the steps of the above-mentioned gesture segmentation and recognition method when executing the computer program stored in the memory.

[0117] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0118] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0119] Embodiment 3:

[0120] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, a gesture segmentation and recognition method described in Example 1 is implemented.

[0121] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned gesture segmentation and recognition method can be implemented.

[0122] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0123] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0124] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A gesture segmentation and recognition method, characterized in that: The steps include: S1. Use the smart data glove to collect gesture data, and use a fixed sliding window to divide the collected gesture data into n continuous window data to obtain a continuous window data set X = {x 1 ,x 2 ,…,x n }, Among them, the i-th window data is marked as x i ; S2. Divide each window data obtained in step S1 into three sub-windows of equal size. Then the i-th window data is divided into three sub-window data, namely S3. For the sub-window data of each window obtained in step S2, the internal similarity features between the sub-windows of each window are calculated using the distance calculation function; S4. Based on the internal similarity features between the sub-windows of each window obtained in step S4, the absolute difference between the minimum value of the internal similarity features in the previous window and the maximum value of the internal similarity features in the current window, and the absolute difference between the minimum value of the internal similarity features in the current window and the maximum value of the internal similarity features in the previous window are calculated, and a method for determining similarity features of adjacent windows is established to determine whether the gesture window is a basic gesture action window or a transition gesture action window, and obtain a basic gesture action window and a transition gesture action window respectively; S5. Upsampling the transition gesture action window obtained in step S4; S6. Input the obtained basic gesture action window and the processed transition gesture action window into the gesture action recognition model for recognition, respectively, to obtain the basic gesture action and the transition gesture action.

2. A gesture segmentation and recognition method according to claim 1, characterized in that: The sensors on the smart data glove in step S1 are a three-axis accelerometer and a three-axis gyroscope. The i-th window data x i ∈R w×C , w is the window size, and C is the number of channels of the sensor.

3. The method for hand gesture segmentation and recognition according to claim 2, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. For the i-th window x i , the internal similarity feature of the i-th window The calculation formula is as follows: in, is the internal similarity feature of the first sub-window of the i-th window, is the internal similarity feature of the second sub-window of the i-th window, is the internal similarity feature of the third sub-window of the i-th window, and d is the Manhattan distance calculation function; the Manhattan distance calculation function is as follows: d(sx p ,sx q )=∑|sx p -sx q | Among them, d(sx p , sx q ) refers to the distance between sub-window p and sub-window q; S3.

2. The Manhattan distance calculation function is calculated using the accelerometer and gyroscope data, that is, sx in the Manhattan distance calculation function p , sx q Use the accelerometer and gyroscope data to replace and obtain the internal similarity features f for the p-th sub-window and the q-th sub-window pq The calculation formula is: Among them, a is the acceleration data, g is the gyroscope data, and the window is divided into three sub-windows. Represents the average acceleration value of the j-axis in the q-th sub-window, It represents the average angular velocity value of the j-axis in the q-th sub-window, the j-axis is the x-coordinate axis, the y-coordinate axis or the z-coordinate axis, p=1 or 2 or 3, q=1 or 2 or 3, and p and q are not equal.

4. The method for hand gesture segmentation and recognition according to claim 3, characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. Set the i-1th window as the previous window, the i-th window as the current window, and calculate the absolute value b between the minimum value of the internal similarity feature in the previous window and the maximum value of the internal similarity feature in the current window i-1 , the calculation formula is: Among them, min is the minimum function and max is the maximum function; S4.

2. Calculate the absolute difference e between the minimum value of the similarity feature within the current window and the maximum value of the similarity feature within the previous window i , the calculation formula is: S4.

3. Establish a method for determining similar features of adjacent windows: S4.3.

1. Set the detection flag representing the transition gesture action window to Tag t , the default is Tag t Set to non-transition gesture, the transition gesture action start threshold is TH ts , the transition gesture end threshold is TH te ; S4.3.

2. In Tag t For non-transition gestures, e i The transition gesture action starts at the threshold TH ts For comparison, if e i Greater than TH ts , the previous window is regarded as the basic gesture action window, and the current window is regarded as the beginning of the transition gesture action window, so the current window is added to the transition gesture action window X tans ; Otherwise, the previous window is considered the transition gesture action window.

5. A gesture segmentation and recognition method according to claim 4, characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. In Tag t In the case of a transition gesture action window, by adding the current window to the buffer X tans To start the aggregation transition window; the aggregation process is to convert b i-1 The transition gesture ends at the threshold TH te For comparison, if b i-1 Above the threshold TH te , it is considered that the end of the transition gesture action window will be reached, and the previous window will be aggregated as the end of the transition gesture action; All detected transition windows are aggregated into X tans , then, for X tans Upsampling is performed to resize it to a fixed window size to match the input size of the recognition model; otherwise, window aggregation continues in the next round; S5.

2. When the process of aggregating the windows exceeds the maximum length of the transition gesture action, it is processed as a static gesture in the basic gesture action; S5.

3. When both windows have mixed samples of transition gestures and basic gestures, the values ​​e of the two adjacent windows will be higher than the threshold transition gesture start threshold TH ts , the latest window is considered to be the beginning of the transition gesture action window.

6. A gesture segmentation and recognition method according to claim 5, characterized in that: The specific implementation method of step S6 includes the following steps: S6.

1. Construct two gesture recognition models, each of which is a CNN-LSTM network model architecture; S6.

2. The obtained basic gesture action window and the processed transition gesture action window are respectively input into the two gesture action recognition models constructed in step S6.1 for recognition to obtain the basic gesture action and the transition gesture action.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a gesture segmentation and recognition method as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the gesture segmentation and recognition method described in any one of claims 1 to 6 is implemented.

Images