Dynamic gesture recognition method, recognition system, recognition equipment and computer readable storage medium

Abstract

The invention relates to a dynamic gesture recognition method, a recognition system, recognition equipment and a computer readable storage medium. The dynamic gesture recognition method comprises the following steps: S1, acquiring a hand image video stream, and extracting and acquiring time sequence hand features based on the video stream; s2, inputting the time sequence hand features into a dynamic gesture recognition model for processing to generate a gesture category, category features and a gesture category probability vector, and comparing the gesture category probability vector with a set threshold to determine a corresponding gesture category and category features; and S3, judging according to the gesture category and the category characteristics, and determining a dynamic gesture. According to the invention, the accuracy of dynamic gesture recognition can be improved based on the monocular camera.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle-mounted systems, and in particular to a dynamic gesture recognition method, a recognition system, a recognition device, and a computer-readable storage medium. Background Art

[0002] The monocular camera deep learning recognition solution mainly relies on the monocular camera to collect image data, and analyzes and processes this data through a deep learning model to achieve gesture recognition. However, this solution has some obvious problems:

[0003] First, single-frame depth estimation is unstable. In the deep learning recognition scheme of a monocular camera, depth features are usually obtained by analyzing and estimating a single-frame image. However, due to the limited information provided by a single-frame image and the influence of environmental factors, single-frame depth estimation is often not accurate and stable enough.

[0004] Secondly, the general gesture deep learning recognition model is based on the single-frame estimation results. This means that the performance of the model depends largely on the accuracy of the single-frame depth estimation. If the single-frame depth estimation is biased, the recognition results of the entire model will also be affected. Moreover, since the model is trained and optimized based on the single-frame estimation results, it may not be able to fully utilize the information on the time series, resulting in suboptimal performance when processing dynamic gestures.

[0005] The monocular camera and depth camera fusion solution attempts to combine the advantages of monocular cameras and depth cameras to improve the accuracy and robustness of gesture recognition. However, this solution also has some disadvantages:

[0006] On the one hand, the design of the fusion of monocular camera and depth camera is difficult. The fusion of two different types of cameras requires solving a series of technical problems, including camera calibration, data fusion algorithm design, system synchronization, etc. These problems increase the difficulty and complexity of system design, requiring more R&D resources and time.

[0007] On the other hand, the cost is significantly increased and the power consumption is increased. Using a depth camera usually increases the cost of the system because the price of a depth camera is relatively high. In addition, using a monocular camera and a depth camera at the same time will also increase the power consumption of the system, which is an important consideration for some mobile devices or application scenarios with high power consumption requirements. Summary of the invention

[0008] In view of the above problems in the prior art, the present invention proposes a dynamic gesture recognition method, a recognition system, a recognition device and a computer-readable storage medium, which can improve the accuracy of dynamic gesture recognition based on a monocular camera.

[0009] Specifically, the present invention proposes a dynamic gesture recognition method, comprising the steps of:

[0010] S1, obtaining a hand image video stream and extracting temporal hand features based on the video stream;

[0011] S2, inputting the time series hand features into a dynamic gesture recognition model for processing to generate a gesture category, category features and a gesture category probability vector, and comparing the gesture category probability vector with a set threshold to determine a corresponding gesture category and category features;

[0012] S3, making a decision based on the gesture category and category features to determine a dynamic gesture.

[0013] According to one embodiment of the present invention, step S2 includes:

[0014] S21, judging whether the hand space feature is valid, if not valid, proceeding to S22, if valid, proceeding to S23;

[0015] S22, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and the process proceeds to S25;

[0016] S23, inputting the hand space feature into a dynamic gesture recognition model to generate a gesture category, category feature and a gesture category probability vector;

[0017] S24, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the largest value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output.

[0018] S25, end.

[0019] According to one embodiment of the present invention, the dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used to extract and upgrade the input time series hand features to obtain The MobileViT module is used to convert the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector

[0020] According to one embodiment of the present invention, the loss function J of the dynamic gesture recognition model is gesture It is composed of CE loss. Assume that in the dataset M, the true category of the i-th sample is C i , loss function J gesture The calculation formula is:

[0021]

[0022] According to one embodiment of the present invention, step S3 includes:

[0023] S31, set the width to W G The time filter window of

[0024] At time t, let the time window slide to obtain a length of W G Gesture category sequence:

[0025]

[0026] in, for or t*in[t-(W G -1),t];

[0027] Get the gesture classification feature sequence at time t:

[0028]

[0029] in, for or NULL;

[0030] S32, determining the number of gesture categories Is the percentage of sequence length less than T? valid , if so, then Go to S34, if not, go to S33;

[0031] S33, yes Sort the gesture categories by quantity to find the gesture category with the most occurrences and multiple gesture categories Based on gesture category to render judgment;

[0032] S34, end.

[0033] According to one embodiment of the present invention, step S33 includes:

[0034] S331, judgment If so, then Enter S34, if not enter S332, Tmax_g is the set threshold for high confidence category judgment;

[0035] S332, judgment If yes, go to S334, if no, go to S333, T min_g is the set threshold for low confidence category judgment;

[0036] S333, judgment If Enter S34;

[0037] S334, based on gesture category And the corresponding gesture category feature vector set The feature vectors of the corresponding categories Calculate the average cosine similarity CS gestureij , the calculation formula is:

[0038]

[0039] Select CS gestureij The gesture category and category features corresponding to the minimum value are taken as the final output

[0040] Selection The feature with the smallest cosine similarity value is Do exponentially weighted updates,

[0041] Enter S34.

[0042] According to one embodiment of the present invention, in step S1, the process of extracting a hand image includes the steps of:

[0043] S11, normalize the image to unify the pixel value scale;

[0044] S12, inputting the normalized image into the YOLO target detection model, and obtaining N hand bounding boxes through the Non-Maximumsuppression algorithm;

[0045] S13, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region;

[0046] S14, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

[0047] The present invention also provides a dynamic gesture recognition system, which is applicable to the above-mentioned dynamic gesture recognition method, and the dynamic gesture recognition system comprises:

[0048] An acquisition unit, configured to acquire a hand image video stream and extract and acquire temporal hand features based on the video stream;

[0049] a processing unit configured to process the temporal hand feature input into a dynamic gesture recognition model to generate a gesture category, a category feature, and a gesture category probability vector;

[0050] a comparison unit, configured to compare the gesture category probability vector with a set threshold to determine a corresponding gesture category and category feature;

[0051] The decision unit is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

[0052] The present invention also provides a dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned dynamic gesture recognition methods when executing the computer program.

[0053] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the dynamic gesture recognition method as described in any one of the above items are implemented.

[0054] The present invention provides a dynamic gesture recognition method, recognition system, recognition device and computer-readable storage medium, which optimize the existing algorithm framework based on a monocular camera, enhance the algorithm feature extraction capability, and thus improve the accuracy of the system's dynamic gesture recognition.

[0055] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are included to provide further explanation of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principle of the present invention.

[0057] In the attached figure:

[0058] Figure 1 A flowchart of a dynamic gesture recognition method according to an embodiment of the present invention is shown.

[0059] Figure 2 A module flow chart for identifying gesture categories and category features according to an embodiment of the present invention is shown.

[0060] Figure 3A simplified layout diagram of a dynamic gesture data collection device according to an embodiment of the present invention is shown.

[0061] Figure 4 A structural schematic diagram of a dynamic gesture recognition system according to an embodiment of the present invention is shown.

[0062] Figure 5 A system block diagram of a dynamic gesture recognition device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0065] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0066] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0067] In the description of the present application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0068] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0069] Figure 1 The flowchart of a dynamic gesture recognition method according to an embodiment of the present invention is shown. As shown in the figure, the present invention provides a dynamic gesture recognition method including the following steps:

[0070] S1, obtain the video stream of the hand image and extract the temporal hand features based on the video stream. Specifically, obtain the video stream as input data. Separate the hand from the complex background to generate a hand image. Extract the hand spatial features based on the hand image. Extract the temporal hand features based on the hand spatial features. After obtaining the hand spatial features, in order to consider the temporal changes of the gesture, it is necessary to extract the temporal hand features based on the hand spatial features.

[0071] S2, process the temporal hand features to generate gesture categories, category features and gesture category probability vectors, and compare the gesture category probability vectors with the set thresholds to determine the corresponding gesture categories and category features. Use a specific gesture recognition module to process these temporal hand features to determine the gesture category and the corresponding category features. The gesture recognition module is a trained deep learning model that can predict different gesture categories and the feature representations corresponding to each category based on the input temporal hand features.

[0072] S3, making a decision based on the gesture category and category features to determine the dynamic gesture.

[0073] In some examples, at step S1, the process of extracting the hand image includes the steps of:

[0074] S11, normalize the image to unify the pixel value scale. Because images from different sources may have different pixel value ranges, the pixel values ​​of the image can be unified to a specific scale through normalization. Define a frame image in the video stream as I t ∈R H*W*C , that is, the height of the image at time t is H, the width is W, and the color channel is C. In this example, the video resolution is 1080P, the color encoding is RGB, and the number of frames per second is 30. Then t can be equivalent to the number of frames (1 frame sampling time is 33ms). The image is normalized, and the corresponding calculation formula is:

[0075] I t_normal =I t / 255.

[0076] S12, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes. YOLO is an efficient and widely used One-Stage target detection algorithm, which is based on a convolutional neural network and can locate and predict the category of the target object in the input image in a single forward propagation process. For hand image extraction, it can quickly detect whether there is a hand in the image and the approximate location information of the hand. The Non-Maximum suppression algorithm will filter the bounding boxes based on the confidence of each bounding box (indicating the possibility that the bounding box contains the target object) and the degree of overlap with other bounding boxes. After being processed by the Non-Maximumsuppression algorithm, N hand bounding boxes are obtained. These bounding boxes relatively accurately define the location of the hands in the image.

[0077] S13, formulate a ROI area (Region of Interest), and filter the target hand on the hand bounding box based on the ROI area. In order to focus on single-hand gestures and ensure that subsequent processing is targeted at a specific target hand, an ROI area needs to be formulated. This ROI area is usually a range pre-set according to specific application scenarios and requirements. It can be a specific rectangular area or other shaped area in the image, which is used to limit the hand range for subsequent processing. By calculating the intersection-and-union (IOU) of each hand bounding box and the ROI area, that is, the area of ​​the overlapping part of the hand bounding box and the ROI area divided by their combined total area, the degree of overlap between the hand bounding box and the region of interest is measured to determine the existence of the target hand. In this example, the ROI is set to the image Area. The corresponding filtering formula is:

[0078]

[0079] S14, if the target hand exists, the filtered hand image is cropped to be used as input for subsequent operation steps. To reduce subsequent computing power consumption, the cropped image is scaled to a resolution and normalized. In this example, the normalization formula for scaling to a resolution of 256*256 is:

[0080]

[0081] If the target hand does not exist, then Is set to the invalid value NULL.

[0082] In some examples, step S2 includes:

[0083] S21, judging whether the hand space feature is valid, if not valid, proceeding to S52, if valid, proceeding to S53;

[0084] S22, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and the process proceeds to S25;

[0085] S23, input the hand space features into the dynamic gesture recognition model to generate gesture categories, category features and gesture category probability vectors. The dynamic gesture recognition model is pre-trained, and identifies the corresponding gesture categories, category features and probability distribution of each gesture category based on the input hand space features, and integrates them into the gesture category probability vector In order to provide a basis for subsequent judgment.

[0086] S24, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the maximum value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output. Set the threshold T g It is a standard set in advance to measure the credibility of the prediction results. If the maximum value is greater than the set threshold T g , indicating that the dynamic gesture recognition model has a high confidence in predicting the current gesture category, so the gesture category and category features corresponding to the maximum probability are selected for output. If the maximum value does not exceed the set threshold, the credibility is low, indicating that the judgment of the current gesture is unreliable, then the gesture category is set to background, the category features are set to invalid values, and then the corresponding results are output.

[0087] S25, end.

[0088] Figure 2 The module flow chart of identifying gesture categories and category features of an embodiment of the present invention is shown. As shown in the figure, the dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used as the backbone network to input the temporal hand features. Perform feature extraction and dimension upgrading to obtain The MobileViT module is a feature enhancement module used to transform the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector

[0089] Specifically, the LSTM module is used to classify the temporal hand features. Position encoding is performed to enable the model to distinguish the order of gestures at different time points, thereby capturing the temporal logic of the action. Extract time series features and increase dimension to obtain The MobileViT module can effectively encode local and global information. The MobileViT module uses Transformer to replace local modeling in convolution with global modeling. This makes the MobileViT module have the properties of CNN (convolutional neural network) and ViT (Vision Transformer), which helps it learn better representations with fewer parameters and simple training methods. Input into the MobileViT module to perform temporal and spatial feature fusion and obtain the feature vector with gesture category Then Input to MLP, MLP (Multi-layer Perceptron) is a basic feedforward neural network that further maps high-dimensional feature vectors to low-dimensional space for more detailed feature classification. The Softmax function converts the output of MLP into a probability distribution to obtain the gesture category probability vector Among them, C o Indicates the number of predefined gesture categories. The median value is the largest and exceeds the set threshold T g That category is the predicted gesture category at the current moment t If no item is satisfied, it is defined as background category Its value is NULL.

[0090] Define the gesture recognition model as M gesture , in use, only Not equal to will be input into M gesture This shows that only when the time-series hand features are detected to be valid at the current moment t, can this moment become a gesture event. At the same time, because M gesture For a multi-stage model, the gesture category feature vector can be obtained from the MobileViT module The dynamic gesture recognition module calling formula is as follows:

[0091]

[0092] Through strict input checking and confidence judgment, the accuracy and reliability of the output results of the dynamic gesture recognition module are guaranteed, enabling it to effectively process various types of hand posture data in actual application scenarios and reduce the occurrence of misjudgment.

[0093] In some examples, the feature dimension C obtained by the LSTM module after processing and upgrading the temporal hand features is u =128. C o =7, C o Represents the number of predefined gesture categories. In this example, there are seven common dynamic gestures, namely, swipe up, swipe down, swing left, swing right, click forward and backward, fist, and release. Set the threshold T g =0.65, used to measure the credibility of each element in the gesture category probability vector output by the MLP&Softmax module. g =768, C g is the dimension of the gesture category feature vector output by the MobileViT module. The 768-dimensional setting reflects the refinement of the MobileViT module in feature learning and representation.

[0094] In some examples, the loss function J of the dynamic gesture recognition model is gesture It is composed of CE loss. Assume that in the dataset M, the true category of the i-th sample is C i , loss function J gesture The calculation formula is:

[0095]

[0096] By traversing all samples, summing up the loss of each sample and taking the average, the performance of the model on the entire dataset is comprehensively evaluated. It is the gesture category probability vector predicted by the model for the i-th sample at time t. It means taking the probability value corresponding to the true category position from the predicted probability vector. The cross entropy loss is calculated for a single sample.

[0097] Since open source data cannot meet the training requirements of the dynamic gesture recognition model, a data collection method suitable for the training requirements is adopted. In some examples, during the training process of the dynamic gesture recognition model, dynamic gesture data is collected as a data set, including the steps of:

[0098] It is equipped with dual-mode cameras with multiple camera angles, and the optical center of the middle camera is facing the person being captured. Figure 3 The following is a simplified layout diagram of a dynamic gesture data collection device according to an embodiment of the present invention. As shown in the figure, in this example, five 1000w pixel dual-mode cameras camera1 to camera5 are configured, and each Meige camera selects a camera angle (-60°, -30°, 0°, 30°, 60°), and the optical center of the middle camera faces the person being collected.

[0099] The collector needs to make C in front of the screen at will. o The dual-mode camera captures the gesture video of each collector. In this example, a green curtain is selected as the background.

[0100] Execute step S1 to extract temporal hand features.

[0101] The characteristics of this data collection method are:

[0102] 1) The same gesture will have different backgrounds in different environments, which greatly interferes with the algorithm. Due to practical conditions, data collection cannot cover all scenes, so a green curtain background is used to facilitate background replacement using matting technology in the later stage.

[0103] 2) Gestures are presented differently under different imaging perspectives. Using multiple camera perspectives to obtain multi-perspective data can make up for problems such as finger occlusion that are prone to occur in a single perspective, enrich the model's cognition of the spatial morphology of gestures, and enhance its generalization ability.

[0104] 3) In different application scenarios, the imaging method and imaging quality are different. For example, in the car cabin, gesture recognition cameras mostly use infrared imaging, while in some conference applications, RGB imaging is mostly used. Therefore, high-resolution, IR / RGB dual-mode digital cameras are used to cover most application scenarios.

[0105] In the actual case, 70 volunteers (male:female ratio 6:4) were recruited to collect C o There are gesture videos, each of which has a duration ranging from 1s to 8s. These data constitute the dynamic gesture dataset of the present invention.

[0106] In some examples, during the training of the dynamic gesture recognition model, the Batch Size is set to 64, which means that the model will process 64 sample data at the same time in each training iteration. The learning rate decay strategy uses the cosine learning rate decrease, and the initial learning rate Lr gesture Set to 0.001, this value determines the step size of the model parameter update at the beginning of training. A total of 150 epochs are trained, which is set after comprehensive consideration of factors such as model complexity and data set size. The purpose is to allow the model to learn as many useful gesture recognition features as possible from the data without overfitting.

[0107] Furthermore, let the video FPS be 30, because the step size of the temporal filter in extracting temporal hand features is 1, and the time t is equivalent to the number of frames. As the video plays, that is, time passes, many or The gesture state sequence GS 3D and by or NULL composed of gesture classification feature sequence FS 3D However, a clear gesture command is required to implement subsequent operations (human-computer interaction, vehicle control, etc.). Therefore, in step S3, a time filter needs to be added again for filtering, and the filter width is W. G , step length S G Equal to W G This means that step S3 outputs t+n*W G The gesture at the moment, and it must satisfy n∈N,t%W G ==0, this setting has the following two reasons:

[0108] Assume that the basic time length of the complete dynamic gesture action is W G , and there is no overlapping area between the gestures. G The filter with the width and step length can accurately capture a complete and independent gesture action;

[0109] The previous processing has fully considered and extracted the time-dependent feature relationship. When the time filter slides in step S3, it does not need to intersect with the sliding position of the previous step. This means that there is no need to repeatedly extract features that are closely related to adjacent moments, but rather a larger span can be used to focus on the complete gesture action, improving processing efficiency.

[0110] In some examples, step S3 includes:

[0111] S31, build time series, set width to W G The time filter window is used to capture the gesture category sequence and gesture classification feature sequence within a certain period of time;

[0112] At time t, let the time window slide to obtain a length of W G Gesture category sequence:

[0113]

[0114] in, for or t * in[t-(W G -1),t];

[0115] Get the gesture classification feature sequence at time t:

[0116]

[0117] in, for or NULL;

[0118] S32, preliminary screening, determining the number of gesture categories Is the percentage of sequence length less than T? valid , if so, then (background category), go to S34, if not, go to S33;

[0119] S33, statistical sorting, Sort the gesture categories by quantity to find the gesture category with the most occurrences and multiple gesture categories Based on gesture category to render judgment;

[0120] S34, end.

[0121] In some examples, step S33 includes:

[0122] S331, high confidence category judgment, judgment If so, then Enter S34, if not, enter S332;

[0123] S332, medium confidence category judgment, judgment If yes, go to S334, if no, go to S333;

[0124] S333, low confidence category judgment, judgment If Enter S34;

[0125] S334, cosine similarity decision, based on gesture category And the corresponding gesture category feature vector set The feature vectors of the corresponding categories Calculate the average cosine similarity CS gestureij , the calculation formula is:

[0126]

[0127] Select CS gestureij The gesture category and category features corresponding to the minimum value are taken as the final output

[0128] Selection The feature with the smallest cosine similarity value is Do exponentially weighted updates,

[0129] Enter S34.

[0130] It should be noted that after the preliminary screening in step S32, sometimes ambiguous gesture judgments may be obtained. This is because the time series characteristics of many gestures are similar, resulting in Based on the inspiration from face registration in face recognition algorithm, K-mean clustering algorithm is used to collect the most representative gesture category feature vectors from the validation set. i=0,1,...,C g -1, C g indivual As the registered feature vector of each category of dynamic gestures, in step S334, the cosine similarity comparison method is used to replace the quantity comparison method to further determine the final dynamic gesture.

[0131] In some examples, W G =66,W G As the width of the time filter window, it defines the range used for comprehensive judgment of gestures in the time dimension. valid =0.8, T valid It is used to measure the proportion of gesture categories in the sequence length. max_g =0.70, T max_g is the set threshold for high confidence category judgment. min_g =0.55, which is the threshold for low confidence category judgment. β = 0.9, β is a weight parameter in the exponential weighted update, which is set to maintain the stability of the feature vector.

[0132] The present invention also provides a dynamic gesture recognition system, which is applicable to the above-mentioned dynamic gesture recognition method. Figure 4 The structure diagram of a dynamic gesture recognition system according to an embodiment of the present invention is shown. As shown in the figure, the dynamic gesture recognition system 400 includes:

[0133] An acquisition unit 401 is configured to acquire a video stream;

[0134] A first extraction unit 402 is configured to extract a hand image based on a video stream;

[0135] A second extraction unit 403 is configured to extract hand spatial features based on the hand image;

[0136] The third extraction unit 404 is configured to extract temporal hand features based on the hand spatial features;

[0137] A gesture recognition unit 405, configured to recognize the temporal hand features to obtain gesture categories and category features;

[0138] The decision unit 406 is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

[0139] Figure 5 The system block diagram of a dynamic gesture recognition device according to an embodiment of the present invention is shown. As shown in the figure, the simulation device 500 may include an internal communication bus 501, a processor 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, and a communication port 505. When applied on a personal computer, the simulation device 500 may also include a hard disk 506. The internal communication bus 501 can realize data communication between the components of the simulation device 500. The processor 502 can make judgments and issue prompts. In some embodiments, the processor 502 can be composed of one or more processors. The communication port 505 can realize data communication between the simulation device 500 and the outside. In some embodiments, the simulation device 500 can send and receive information and data from the network through the communication port 505. The simulation device 500 may also include different forms of program storage units and data storage units, such as a hard disk 506, a read-only memory (ROM) 503 and a random access memory (RAM) 504, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 502. The processor 502 executes these instructions to implement the main part of the method. The result processed by the processor 502 is transmitted to the user device through the communication port 505 and displayed on the user interface.

[0140] The above-mentioned simulation method can be implemented as a computer program, stored in the hard disk 506, and loaded into the processor 502 for execution to implement the simulation method of the present application.

[0141] The present invention also provides a dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the steps of the aforementioned dynamic gesture recognition method when executing the computer program.

[0142] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned dynamic gesture recognition methods are implemented.

[0143] The specific implementation methods and technical effects of the dynamic gesture recognition device and the computer-readable storage medium can be found in the above-mentioned embodiments of the detection method provided by the present invention, which will not be described in detail here.

[0144] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0145] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0146] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0147] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0148] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention covers modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A dynamic gesture recognition method, comprising the steps of: S1, obtaining a hand image video stream and extracting temporal hand features based on the video stream; S2, inputting the time series hand features into a dynamic gesture recognition model for processing to generate a gesture category, category features and a gesture category probability vector, and comparing the gesture category probability vector with a set threshold to determine a corresponding gesture category and category features; S3, making a decision based on the gesture category and category features to determine a dynamic gesture.

2. The dynamic gesture recognition method according to claim 1, characterized in that: Step S2 includes: S21, judging whether the hand space feature is valid, if not valid, proceeding to S22, if valid, proceeding to S23; S22, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and then the process goes to S25; S23, inputting the hand space feature into a dynamic gesture recognition model to generate a gesture category, category feature and a gesture category probability vector; S24, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the largest value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output. S25, end.

3. The dynamic gesture recognition method according to claim 1 or 2, characterized in that: The dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used to extract and upgrade the input time series hand features to obtain The MobileViT module is used to convert the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector 4. The dynamic gesture recognition method according to claim 3, characterized in that: The loss function J of the dynamic gesture recognition model gesture It is composed of CE loss. Assume that in the dataset M, the true category of the i-th sample is C i , loss function J gesture The calculation formula is:

5. The dynamic gesture recognition method according to claim 1, characterized in that: Step S3 includes: S31, set the width to W G The time filter window of At time t, let the time window slide to obtain a length of W G Gesture category sequence: in, for or t * in[t-(W G -1),t]; Get the gesture classification feature sequence at time t: in, for or NULL; S32, determining the number of gesture categories Is the percentage of sequence length less than T? valid , if so, then Go to S34, if not, go to S33; S33, yes Sort the gesture categories by quantity to find the gesture category with the most occurrences and multiple gesture categories Based on gesture category to render judgment; S34, end.

6. The dynamic gesture recognition method according to claim 5, characterized in that: Step S33 includes: S331, judgment If so, then Enter S34, if not enter S332, T max_g is the set threshold for high confidence category judgment; S332, judgment If yes, go to S334, if no, go to S333, T min_g is the set threshold for low confidence category judgment; S333, judgment If Enter S34; S334, based on gesture category And the corresponding gesture category feature vector set The feature vectors of the corresponding categories Calculate the average cosine similarity The calculation formula is: Select The gesture category and category features corresponding to the minimum value are taken as the final output Selection The feature with the smallest cosine similarity value is Do exponentially weighted updates, Enter S34.

7. The dynamic gesture recognition method according to claim 1, characterized in that: In step S1, the process of extracting the hand image includes the steps of: S11, normalize the image to unify the pixel value scale; S12, inputting the normalized image into the YOLO target detection model, and obtaining N hand bounding boxes through the Non-Maximumsuppression algorithm; S13, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region; S14, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

8. A dynamic gesture recognition system, applicable to the dynamic gesture recognition method according to claim 1, characterized in that: The dynamic gesture recognition system comprises: An acquisition unit, configured to acquire a hand image video stream and extract and acquire temporal hand features based on the video stream; a processing unit configured to process the temporal hand feature input into a dynamic gesture recognition model to generate a gesture category, a category feature, and a gesture category probability vector; a comparison unit, configured to compare the gesture category probability vector with a set threshold to determine a corresponding gesture category and category feature; The decision unit is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

9. A dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the dynamic gesture recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic gesture recognition method according to any one of claims 1 to 7 are implemented.

Images