Dynamic gesture recognition method, recognition system, recognition equipment and computer readable storage medium

Abstract

The invention relates to a dynamic gesture recognition method, a recognition system, recognition equipment and a computer readable storage medium. The dynamic gesture recognition method comprises the following steps: S1, acquiring a video stream; s2, extracting a hand image based on the video stream; s3, extracting hand space features based on the hand image; s4, extracting time sequence hand features based on the hand space features; s5, recognizing the time sequence hand features to obtain gesture categories and category features; and S6, performing judgment according to the gesture category and the category characteristics, and determining a dynamic gesture. According to the invention, the accuracy of dynamic gesture recognition can be improved based on the monocular camera.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle-mounted systems, and in particular to a dynamic gesture recognition method, a recognition system, a recognition device, and a computer-readable storage medium. Background Art

[0002] The monocular camera deep learning recognition solution mainly relies on the monocular camera to collect image data, and analyzes and processes this data through a deep learning model to achieve gesture recognition. However, this solution has some obvious problems:

[0003] First, single-frame depth estimation is unstable. In the deep learning recognition scheme of a monocular camera, depth features are usually obtained by analyzing and estimating a single-frame image. However, due to the limited information provided by a single-frame image and the influence of environmental factors, single-frame depth estimation is often not accurate and stable enough.

[0004] Secondly, the general gesture deep learning recognition model is based on the single-frame estimation results. This means that the performance of the model depends largely on the accuracy of the single-frame depth estimation. If the single-frame depth estimation is biased, the recognition results of the entire model will also be affected. Moreover, since the model is trained and optimized based on the single-frame estimation results, it may not be able to fully utilize the information on the time series, resulting in suboptimal performance when processing dynamic gestures.

[0005] The monocular camera and depth camera fusion solution attempts to combine the advantages of monocular cameras and depth cameras to improve the accuracy and robustness of gesture recognition. However, this solution also has some disadvantages:

[0006] On the one hand, the design of the fusion of monocular camera and depth camera is difficult. The fusion of two different types of cameras requires solving a series of technical problems, including camera calibration, data fusion algorithm design, system synchronization, etc. These problems increase the difficulty and complexity of system design, requiring more R&D resources and time.

[0007] On the other hand, the cost is significantly increased and the power consumption is increased. Using a depth camera usually increases the cost of the system because the price of a depth camera is relatively high. In addition, using a monocular camera and a depth camera at the same time will also increase the power consumption of the system, which is an important consideration for some mobile devices or application scenarios with high power consumption requirements. Summary of the invention

[0008] In view of the above problems in the prior art, the present invention proposes a dynamic gesture recognition method, a recognition system, a recognition device and a computer-readable storage medium, which can improve the accuracy of dynamic gesture recognition based on a monocular camera.

[0009] Specifically, the present invention proposes a dynamic gesture recognition method, comprising the steps of:

[0010] S1, obtain video stream;

[0011] S2, extracting a hand image based on the video stream;

[0012] S3, extracting hand spatial features based on the hand image;

[0013] S4, extracting temporal hand features based on the hand spatial features;

[0014] S5, identifying the temporal hand features to obtain gesture categories and category features;

[0015] S6, making a decision based on the gesture category and category features to determine a dynamic gesture.

[0016] According to one embodiment of the present invention, in step S2, the process of extracting the hand image includes the steps of:

[0017] S21, normalize the image to unify the pixel value scale;

[0018] S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes;

[0019] S23, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region;

[0020] S24, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

[0021] According to one embodiment of the present invention, in step S3, the process of extracting hand space features includes the following steps:

[0022] S31, determine whether the hand image is valid, if not, proceed to S32, if valid, proceed to S33;

[0023] S32, setting the hand space feature to an invalid value, outputting the hand space feature, and entering S34;

[0024] S33, inputting the hand image into a 3D feature extraction model to extract hand spatial features, where the hand spatial features include 2D features and depth features;

[0025] S34, end.

[0026] According to one embodiment of the present invention, in step 33, the 2D features of the hand image are obtained by using a 3D feature extraction model. and the deep feature Z t ;

[0027] 2D features Convert to relative coordinate 2D features in image coordinate system The depth feature Z t Normalized to deep feature Z t_normal .

[0028] According to one embodiment of the present invention, the depth feature The maximum distance of the target relative to the camera is Z max , the minimum distance is Z min .

[0029] According to one embodiment of the present invention, the 3D feature extraction model includes an HourglassNet module, a GraphCNN module and a LFFM module. The HourglassNet module is used to perform feature extraction on the input hand image to obtain the 2D features. The GraphCNN module is used to obtain the vertices of the three-dimensional hand mesh to obtain the deep features. The LFFM module is used to fuse the HourglassNet module and the GraphCNN module, and processes the output of the HourglassNet module to adapt to the input of the GraphCNN module.

[0030] According to one embodiment of the present invention, in the HourglassNet module, a lightweight SCAB operator is used to replace the Residual-Block operator to reduce computing power consumption.

[0031] According to an embodiment of the present invention, after acquiring the hand spatial features in step S33, a feature filtering region is set, and features are further refined within the feature filtering region based on a spatial filtering method.

[0032] According to one embodiment of the present invention, with the current prediction position as the center, an R*R region is selected as the feature filtering region, and a Gaussian filter is used to further refine features within the feature filtering region. The two-dimensional Gaussian distribution formula is:

[0033]

[0034] Where R = 3, σ = 1.5, (x, y) is the position in the image coordinate system, μ x , μ y is the current predicted position.

[0035] According to one embodiment of the present invention, the loss function J of the 3D feature extraction model is3D Including heat map loss J stage and depth loss J depth , the heat map loss J stage It consists of 3 stages, and the loss function J 3D The calculation formula is:

[0036]

[0037] Among them, α is the heat map loss penalty factor, and β is the multi-loss control factor.

[0038] According to one embodiment of the present invention, during the training process of the 3D feature extraction model, the Batch Size is set to 128 and the initial learning rate Lr 3D It is set to 0.001, decayed by 0.1 every 30 epochs of training, and a total of 100 epochs are trained. The datasets use STB and RHD.

[0039] According to one embodiment of the present invention, step S4 includes:

[0040] Set the width to W T The time filter window is aligned with the rightmost side of the time filter to the current time t;

[0041] Construct time series features. At time t, let the time window slide to obtain a length of W T Gesture time series features:

[0042]

[0043] The step size S of the temporal filter T is 1.

[0044] According to one embodiment of the present invention, step S5 includes:

[0045] S51, determining whether the hand space feature is valid, if not valid, proceeding to S52, if valid, proceeding to S53;

[0046] S52, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and the process proceeds to S55;

[0047] S53, inputting the hand space feature into a dynamic gesture recognition model to generate a gesture category, category feature and a gesture category probability vector;

[0048] S54, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the largest value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output.

[0049] S55, end.

[0050] According to one embodiment of the present invention, the dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used to extract and upgrade the input time series hand features to obtain The MobileViT module is used to convert the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector

[0051] According to one embodiment of the present invention, the loss function J of the dynamic gesture recognition model is gesture It is composed of CE loss. Assume that in the dataset M, the true category of the i-th sample is C i , loss function K gesture The calculation formula is:

[0052]

[0053] According to one embodiment of the present invention, during the training process of the dynamic gesture recognition model, the dynamic gesture data is collected as a data set, including the steps of:

[0054] Equipped with dual-mode cameras with multiple camera angles, the optical center of the middle camera faces the subject;

[0055] The collector needs to make C in front of the screen at will. o kinds of gestures, collect gesture videos of each collector;

[0056] Steps S2 to S4 are executed to extract temporal hand features.

[0057] According to one embodiment of the present invention, during the training process of the dynamic gesture recognition model, the batch size is set to 64, the learning rate decay strategy adopts the cosine learning rate decrease, and the initial learning rate Lr gesture The training was performed for 150 epochs.

[0058] According to one embodiment of the present invention, step S6 includes:

[0059] S61, set the width to W GThe time filter window of

[0060] At time t, let the time window slide to obtain a length of W G Gesture category sequence:

[0061]

[0062] in, for or t * in[t-(W G -1),t];

[0063] Get the gesture classification feature sequence at time t:

[0064]

[0065] in, for or NULL;

[0066] S62, determining the number of gesture categories Is the percentage of sequence length less than T? valid , if so, then Go to S68, if not, go to S63;

[0067] S63, yes Sort the gesture categories by quantity to find the gesture category with the most occurrences and multiple gesture categories

[0068] S64, Judgment If so, then Go to S68, if not go to S65;

[0069] S65, Judgment If yes, go to S67, if no, go to S66;

[0070] S66, Judgment If Enter S68;

[0071] S67, based on gesture category And the corresponding gesture category feature vector set The feature vectors of the corresponding categories Calculate the average cosine similarity CS gestireij , the calculation formula is:

[0072]

[0073] Select CS gestureij The gesture category and category features corresponding to the minimum value are taken as the final output

[0074] Selection The feature with the smallest cosine similarity value is Do exponentially weighted updates,

[0075] S68, end.

[0076] The present invention also provides a dynamic gesture recognition system, which is applicable to the above-mentioned dynamic gesture recognition method, and the dynamic gesture recognition system comprises:

[0077] An acquisition unit, configured to acquire the video stream;

[0078] A first extraction unit, configured to extract a hand image based on the video stream;

[0079] a second extraction unit configured to extract hand spatial features based on the hand image;

[0080] A third extraction unit is configured to extract a temporal hand feature based on the hand spatial feature;

[0081] a gesture recognition unit configured to recognize the temporal hand features to obtain gesture categories and category features;

[0082] The decision unit is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

[0083] The present invention also provides a dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned dynamic gesture recognition methods when executing the computer program.

[0084] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the dynamic gesture recognition method as described in any one of the above items are implemented.

[0085] The present invention provides a dynamic gesture recognition method, recognition system, recognition device and computer-readable storage medium, which optimize the existing algorithm framework based on a monocular camera, enhance the algorithm feature extraction capability, and thus improve the accuracy of the system's dynamic gesture recognition.

[0086] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings are included to provide further explanation of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principle of the present invention.

[0088] In the attached figure:

[0089] Figure 1 A flowchart of a dynamic gesture recognition method according to an embodiment of the present invention is shown.

[0090] Figure 2 A schematic diagram showing the distribution of joint points of a hand 21 according to an embodiment of the present invention is shown.

[0091] Figure 3 A module flow chart of hand space feature extraction according to an embodiment of the present invention is shown.

[0092] Figure 5 A schematic structural diagram of the SCAB operator in the HourglassNet module according to an embodiment of the present invention is shown.

[0093] Figure 4 A test set accuracy data graph of the LFFM module according to an embodiment of the present invention is shown.

[0094] Figure 6 A module flow chart for identifying gesture categories and category features according to an embodiment of the present invention is shown.

[0095] Figure 7 A simplified layout diagram of a dynamic gesture data collection device according to an embodiment of the present invention is shown.

[0096] Figure 8 A structural schematic diagram of a dynamic gesture recognition system according to an embodiment of the present invention is shown.

[0097] Fig. 9 A system block diagram of a dynamic gesture recognition device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0098] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0099] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0100] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0101] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0102] In the description of the present application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0103] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0104] Figure 1 The flowchart of a dynamic gesture recognition method according to an embodiment of the present invention is shown. As shown in the figure, the present invention provides a dynamic gesture recognition method including the following steps:

[0105] S1, obtain video stream. Obtain video stream as input data. The video stream can come from various devices, such as cameras, video files, etc., as raw data for subsequent processing. For example, in a vehicle system, the vehicle camera can capture the user's hand movements in real time and generate a video stream.

[0106] S2, extracting hand images based on the video stream. This step is used to detect whether the hand exists, separate the hand from the complex background, and generate a hand image for subsequent processing.

[0107] S3, extracting hand spatial features based on the hand image. Hand spatial features usually include information such as hand shape, joint positions, finger postures, etc. This step extracts hand spatial features through a neural network architecture.

[0108] S4, extracting temporal hand features based on hand spatial features. After obtaining the hand spatial features, in order to consider the temporal changes of the gesture, it is necessary to extract temporal hand features based on the hand spatial features.

[0109] S5, identifying the temporal hand features to obtain gesture categories and category features. A specific gesture recognition module is used to identify these temporal hand features to determine the category of the gesture and the corresponding category features. The gesture recognition module is a trained deep learning model that can predict different gesture categories and the feature representation corresponding to each category based on the input temporal hand features.

[0110] S6, making a decision based on the gesture category and category features to determine the dynamic gesture.

[0111] In some examples, at step S2, the process of extracting the hand image includes the steps of:

[0112] S21, normalize the image to unify the pixel value scale. Because images from different sources may have different pixel value ranges, the pixel values ​​of the image can be unified to a specific scale through normalization. Define a frame image in the video stream as I t ∈R H*W*C , that is, the height of the image at time t is H, the width is W, and the color channel is C. In this example, the video resolution is 1080P, the color encoding is RGB, and the number of frames per second is 30. Then t can be equivalent to the number of frames (1 frame sampling time is 33ms). The image is normalized, and the corresponding calculation formula is:

[0113] I t_normal =I t / 255.

[0114] S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes. YOLO is an efficient and widely used One-Stage target detection algorithm, which is based on a convolutional neural network and can locate and predict the category of the target object in the input image in a single forward propagation process. For hand image extraction, it can quickly detect whether there is a hand in the image and the approximate position information of the hand. The Non-Maximum suppression algorithm will filter the bounding boxes based on the confidence of each bounding box (indicating the possibility that the bounding box contains the target object) and the degree of overlap with other bounding boxes. After being processed by the Non-Maximumsuppression algorithm, N hand bounding boxes are obtained. i=1,...N, these bounding boxes relatively accurately define the position of the hand in the image.

[0115] S23, formulate a ROI area (Region of Interest), and filter the target hand on the hand bounding box based on the ROI area. In order to focus on single-hand gestures and ensure that subsequent processing is targeted at a specific target hand, an ROI area needs to be formulated. This ROI area is usually a pre-set range based on specific application scenarios and requirements. It can be a specific rectangular area or other shaped area in the image, which is used to limit the hand range for subsequent processing. By calculating the intersection-and-union (IOU) of each hand bounding box and the ROI area, that is, the area of ​​the overlapping part of the hand bounding box and the ROI area divided by their combined total area, the degree of overlap between the hand bounding box and the region of interest is measured to determine the existence of the target hand. In this example, the ROI is set to the image Area. The corresponding filtering formula is:

[0116]

[0117] S24, if the target hand exists, the filtered hand image is cropped to be used as input for subsequent operation steps. To reduce subsequent computing power consumption, the cropped image is scaled to a resolution and normalized. In this example, the normalization formula for scaling to a resolution of 256*256 is:

[0118]

[0119] If the target hand does not exist, then Is set to the invalid value NULL.

[0120] In some examples, in step S3, the process of extracting hand spatial features includes the steps of:

[0121] S31, determine whether the hand image is valid, if not, enter S32, if valid, enter S33. At the beginning of the entire extraction of hand space features, it is necessary to first judge the validity of the input hand image. This validity judgment may be based on a variety of factors, such as image integrity, image clarity, and whether it does contain recognizable hand content. If it is found through detection that the hand image does not meet the relevant requirements, that is, it is judged to be invalid, then enter step S32 for corresponding processing; if the hand image passes the validity test and is judged to be a valid image, enter S33 and continue the subsequent feature extraction process.

[0122] S32, set the hand space feature to an invalid value, output the hand space feature, and enter S34;

[0123] S33, input the hand image into a 3D feature extraction model to extract hand spatial features, where the hand spatial features include 2D features and depth features. The 2D features mainly reflect information such as the shape and joint positions of the hand on the image plane. Figure 2 The figure shows the distribution diagram of the 21 joints of the hand according to an embodiment of the present invention. As shown in the figure, the 21 joints of the hand are marked 0 to 20 respectively, the root node is the wrist joint 0, and then 4 joints are used to represent the position of each finger (thumb, index finger, middle finger, ring finger and pinky finger). 2D features It can depict the general shape and posture of the hand on a two-dimensional plane. t It reflects the distance information of each joint of the hand relative to the camera, through the depth feature Z t Combined with 2D features, it can fully describe the state of the hand in three-dimensional space, which plays a key role in subsequent tasks such as accurate recognition of dynamic gestures. and hand space features It is expressed as:

[0124]

[0125] S34, end.

[0126] In some examples, 2D features of the hand image are obtained by a 3D feature extraction model. and the deep feature Z t ; Then the 2D features Convert to relative coordinate 2D features in image coordinate system The deep features Normalized to deep feature Z t_normal .

[0127] It should be noted that 2D features Located in the image coordinate system, and the depth feature Located in the camera coordinate system, the two coordinate systems are different. Because the 2D features here It is only used to express the relative position relationship of the finger under a certain plane projection. It is easier to obtain it under the image coordinate system. Therefore, it is necessary to convert the 2D feature Relative coordinates in the image coordinate system Deep Features Use the predefined target to normalize the distance relative to the camera, unify the information of different dimensions into a similar numerical range, and obtain the depth feature Z t_normal .

[0128] In some examples, deep features The maximum distance of the target relative to the camera is Z max , the minimum distance is Z min .

[0129] Figure 3A module flow chart of hand space feature extraction according to an embodiment of the present invention is shown. As shown in the figure, in some examples, the 3D feature extraction model includes an HourglassNet module, a GraphCNN module and a LFFM module. The HourglassNet module serves as the backbone network. The network structure is shaped like an hourglass, and repeatedly uses Top-down and Bottom-up to infer the position of the joints. The HourglassNet module is used to extract features from the input hand image and obtain 2D features. The GraphCNN module is used to obtain the vertices of the three-dimensional hand mesh, and then derive deep features from them. Specifically, it abstracts the hand into a graph structure consisting of multiple vertices and edges, where the vertices correspond to the key parts of the hand (such as joints, etc.), and the edges represent the connection relationship between these parts, and then extracts deep features from the relationship attributes such as the coordinates of the three-dimensional hand mesh vertices. The LFFM module is used to fuse the HourglassNet module and the GraphCNN module, and it processes the output of the HourglassNet module to adapt to the input of the GraphCNN module. Specifically, the LFFM module is composed of a standard MobileNet, and the SE (Squeeze and Extraction) attention mechanism is added to the last Point-wise convolution stage of each Inverted Bottleneck to enhance the channel dimension features. Secondly, the downsampling operation is replaced by Focus, which is a slicing operation on the feature map. In a feature map, a value is obtained for every other pixel, which is similar to neighboring downsampling. In this way, 4 similar and complementary new feature maps are obtained, which better preserves the integrity of the hand image features, but without information loss. In this way, the W and H information are concentrated in the channel space, and the input channel is expanded by 4 times. Finally, a 2x downsampled feature map without information loss is obtained, which provides a basis for subsequent gesture recognition. Suppose the input feature map F in ∈R C*H*W , and its final output F out ∈R 4C**2W The pseudo code is as follows:

[0130] Patch1=F in [:,∶:2,∶:2];

[0131] Patch2=F in [:,1::2,∶:2];

[0132] Patch3=F in [:,∶:2,1::2];

[0133] Patch4=F in [:,1::2,1::2];

[0134] F out =Concat(Patch1,Patch2,Patch 3, Patch4,axis=0).

[0135] Figure 4 The test set accuracy data graph of the LFFM module of an embodiment of the present invention is shown. As shown in the figure, it can be clearly observed that the curve with the LFFM module is higher than the curve without the LFFM module in most cases. This shows that under the same conditions, the LFFM module can improve the accuracy of the model. Especially in the right half of the curve, the gap between the two gradually increases, which shows that as the data processing proceeds, the advantages of the LFFM module become more obvious and can significantly improve the performance of the model.

[0136] In some examples, in the HourglassNet module, the lightweight SCAB operator is used to replace the Residual-Block operator to reduce computing power consumption. Figure 5 The structural schematic diagram of the SCAB operator in the HourglassNet module of an embodiment of the present invention is shown. As shown in the figure, the operation starts from the Channel Split module at the top, and the input feature map is split along the channel dimension. Generally, this segmentation can divide the input hand feature map into two parts, one part is used for subsequent transformation operations, and the other part is used to be directly processed by the Identity Mapping (residual connection) module. The segmented part of the feature map is first subjected to a 1x1 convolution operation, which is usually used to adjust the number of channels. This is followed by batch normalization (BN) and SiLU activation function processing, which are used to normalize and nonlinearly transform the features. The hand feature map after 1x1 convolution and activation function is then subjected to a 3x3 depth separable convolution (DWConv). Depth separable convolution decomposes the standard convolution operation into depth convolution and point-by-point convolution, which is beneficial to reduce the amount of calculation. It is then processed by batch normalization and SiLU activation function. Then, enter the SE module. The SE module is used to perform attention mechanism operations on the channel dimension. It can adaptively adjust the channel weights, enhance important channel features, and suppress unimportant channel features. The feature map output from the SE module is again subjected to a 1x1 convolution operation to further adjust the number of channels. The hand feature map that has undergone the above series of transformation operations is concatenated with another part of the hand feature map that is directly processed by Identity Mapping in Concat. This concatenation operation merges the two parts of the feature map together in the channel dimension. The concatenated hand feature map is then subjected to a 3x3 depth-separable convolution to further extract features. Finally, batch normalization and SiLU activation function processing are performed to obtain the final output hand feature map.

[0137] It is easy to understand that a 3*3 Fully Conv can be replaced by a Point-wise convolution, a 3*3 Depth-wise convolution, and a Point-wise convolution. Assume that the input hand feature map F in ∈R C*H*W , the size of the output feature map is the same as the input, and the downsampling factor of the Residual-Block operator is 0.5. Then the computing power consumption MAC (Multiply-Add Count) of the Residual-Block operator is:

[0138] MAC Res =6C 2 HW;

[0139] When the SCAB operator expansion factor t is set to 1, its computing power consumption is:

[0140] MAC SCAB =3C 2 HW+9CHW;

[0141] When C=32, the ratio of the two is:

[0142]

[0143] In this way, replacing the Residual-Block operator with the SCAB operator can reduce computing power consumption by about 45%. In addition, SCAB also introduces the concepts of Channel Splitting and Attention (attention mechanism) to further enhance feature fusion capabilities.

[0144] Furthermore, the HourglassNet module is composed of 3 Stage HourglassModules. HourglassModules usually have a unique network architecture, which includes downsampling and upsampling processes. This multi-stage HourglassModule structure helps to extract features of hand images from different levels and scales, making the extracted features more comprehensive and rich. The output calculation module of the HourglassNet module uses Residual-Block plus Point-wise convolution. In the training phase, the heat maps of all stages are involved in the training. In the inference phase, only the heat map of the last stage is output and the ArgMax operation is performed to obtain 2D features. The ArgMax operation is used to find the position with the largest response intensity in the heat map, which usually corresponds to the key feature point of the hand. In this way, the 2D features of the hand can be quickly and accurately obtained in the inference stage. The LLFM module uses the last Stage feature map of HourglassNet as input to obtain the fusion feature, and the GraphCNN module uses this as input to obtain the deep feature Z. t_normal This combination of HourglassNet, LFFM, and GraphCNN makes the entire computational process an end-to-end model, which is beneficial to model training and running speed. Let the 3D feature extraction model be M 3D , the input is the original hand image The 3D features of the hand The extraction can be characterized as:

[0145]

[0146] Among them, reference Figure 3 , only when the normalized hand image Valid, that is, only when a hand is detected in step S2, the normalized image will be input into the 3D feature extraction model M 3D When the hand image If invalid, skip M 3D , set the hand space feature to an invalid value.

[0147] In some examples, after obtaining the spatial features of the hand through the 3D feature extraction model in step S33, the feature filtering area is set, and the features are further refined in the feature filtering area based on the spatial filtering method. Considering that no matter how the 3D feature extraction model is optimized, the predicted position will always have a certain deviation from the actual position, and it may be unreasonable to use 3D data at a single position. Therefore, we assume that the distribution of the actual predicted position satisfies the two-dimensional Gaussian distribution. In practical applications, a method based on spatial filtering is proposed to further extract features. That is, with the current predicted position as the center, the R*R area is taken as the feature filtering area. In this example, R=3, and the feature filter uses a two-dimensional Gaussian filter. The Gaussian filter is a linear smoothing filter based on the Gaussian function, which can weightedly calculate the eigenvalues ​​in the filtering area according to the distance from the center position. The spatial features can be obtained. The two-dimensional Gaussian distribution formula is as follows:

[0148]

[0149] Among them, x, y are the positions in the image coordinate system, μ x ,μ yis the current predicted position. The variable σ determines the width of the Gaussian distribution, that is, it controls the degree of weight attenuation at different positions in the filter area. In this example, σ = 1.5.

[0150] like invalid, Assigned to That is, the values ​​of 2D features and depth features are both -1, which indicates that the key point coordinates exceed the image coordinate system and the hand is located behind the optical center of the camera, which means that the hand space feature is set to an invalid value. This assignment method can be used to identify and exclude invalid feature data in subsequent processing.

[0151] In some examples, the loss function J of the 3D feature extraction model during the training phase is 3D Including heat map loss J stage and depth loss J depth , heat map loss J stage It consists of 3 stages, each of which generates a heat map during the training phase. Heat maps are usually used to represent the response strength of hand features at different positions. Each stage's heat map has its corresponding loss, which reflects the accuracy of the model's prediction of the heat map at each stage. depth It reflects the accuracy of the model in predicting the depth features of the hand. Loss function J 3D The calculation formula is:

[0152]

[0153] Among them, α is the heat map loss penalty factor, which is used to adjust the weight of the heat map loss in the total loss. The heat map loss of different stages is adjusted by α. stage β is a multi-loss control factor, which is used to adjust the weight of depth loss in the total loss. β is used to balance the contribution of heat map loss and depth loss in the total loss function. In this example, α1=α2=0.5, α3=1.0, and β=1.3 can be set.

[0154] In some examples, during the training of the 3D feature extraction model, Batch Size represents the number of samples processed in each training iteration and is set to 128; the initial learning rate Lr 3D The step size of the model parameter update in the iteration, Lr 3DSet to 0.001; decay by 0.1 every 30 epochs of training, that is, after every 30 complete dataset traversals, the learning rate will become 0.1 times the original; a total of 100 epochs are trained, and one epoch means that the model completely traverses the dataset once. The dataset uses STB (Stereo Hand Pose Tracking Benchmark) and RHD (Rendered Hand Pose Dataset). The dataset STB is a real-world dataset containing 18,000 images, including the ground truth of 21 3D hand joint positions and the corresponding depth images. The dataset is divided into 15,000 training samples and 3,000 test samples. In order to make this setting consistent with the RHD dataset, refer to Figure 2 , moving the root joint from the center of the palm to the wrist. RHD is a synthetic dataset containing 41,258 training images and 2,728 test images. During training, the image enhancement method selects conventional random cropping, random scaling, and color domain changes. By adopting these conventional image enhancement methods in training, the diversity of training data can be effectively expanded, the robustness and generalization ability of the model can be improved, and the 3D feature extraction model can be helped to better learn the features of hand images, thereby improving the performance of subsequent gesture recognition.

[0155] In some examples, step S4 includes:

[0156] Set the width to W T The time filter window is used to align the rightmost side of the time filter to the current time t. Existing dynamic gesture recognition work usually takes single-frame spatial features as input and sends them to the gesture recognition module to obtain gesture categories, without considering the temporal correlation of hand features. In actual application scenarios, gestures are recorded in video streams. Therefore, the result at a certain time t should depend on or be included in the features of the previous time, that is, the state at the current time can only be affected by past events. Therefore, a width of W is designed. T The time filter window. Unlike the spatial filter, the center position of the time filter is not at the current time t, because it is impossible to know the event at the next time t+1 in advance, and the event at the current time t can only be estimated based on the events at the historical time. Therefore, the rightmost side of the filter is aligned to the current time t. This shows that the time filter window is based on the current time, and extends a certain time range to the left to cover the relevant time information in the past. In other words, the gesture at the current time t depends on the previous W. T The gesture feature at each moment is represented as a time series width W T characteristics.

[0157] Construct time series features. At time t, let the time window slide to obtain a length of W T Gesture time series features:

[0158]

[0159] Set the step size S of the temporal filter T is 1. Each time the time filter window is slid, it moves only one time unit. A smaller step size can more finely capture the detailed features of the gesture that changes continuously in time, ensuring that the subtle changes between adjacent moments are not missed. The time filter window will evenly pass through each moment, and this collection process can be regarded as a FIFOQueue (First in first out). For example, collecting Need to pop up join in In this example, W T =11.

[0160] In some examples, step S5 includes:

[0161] S51, judging whether the hand space feature is valid, if not valid, proceeding to S52, if valid, proceeding to S53;

[0162] S52, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and the process proceeds to S55;

[0163] S53, input the hand space features into the dynamic gesture recognition model to generate gesture categories, category features and gesture category probability vectors. The dynamic gesture recognition model is pre-trained, and identifies the corresponding gesture categories, category features and probability distribution of each gesture category based on the input hand space features, and integrates them into the gesture category probability vector In order to provide a basis for subsequent judgment.

[0164] S54, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the maximum value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output. Set the threshold T g It is a standard set in advance to measure the credibility of the prediction results. If the maximum value is greater than the set threshold T g, indicating that the dynamic gesture recognition model has a high confidence in predicting the current gesture category, so the gesture category and category features corresponding to the maximum probability are selected for output. If the maximum value does not exceed the set threshold, the credibility is low, indicating that the judgment of the current gesture is unreliable, then the gesture category is set to background, the category features are set to invalid values, and then the corresponding results are output.

[0165] S55, end.

[0166] Figure 6 The module flow chart of identifying gesture categories and category features of an embodiment of the present invention is shown. As shown in the figure, the dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used as the backbone network to input the temporal hand features. Perform feature extraction and dimension upgrading to obtain The MobileViT module is a feature enhancement module used to transform the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector

[0167] Specifically, the LSTM module is used to classify the temporal hand features. Position encoding is performed to enable the model to distinguish the order of gestures at different time points, thereby capturing the temporal logic of the action. Extract time series features and increase dimension to obtain The MobileViT module can effectively encode local and global information. The MobileViT module uses Transformer to replace local modeling in convolution with global modeling. This makes the MobileViT module have the properties of CNN (convolutional neural network) and ViT (Vision Transformer), which helps it learn better representations with fewer parameters and simple training methods. Input into the MobileViT module to perform temporal and spatial feature fusion and obtain the feature vector with gesture category Then Input to MLP, MLP (Multi-layer Perceptron) is a basic feedforward neural network that further maps high-dimensional feature vectors to low-dimensional space for more detailed feature classification. The Softmax function converts the output of MLP into a probability distribution to obtain the gesture category probability vector Among them, C o Indicates the number of predefined gesture categories. The median value is the largest and exceeds the set threshold T g That category is the predicted gesture category at the current moment t If no item is satisfied, it is defined as background category Its value is NULL.

[0168] Define the gesture recognition model as M gesture , in use, only Not equal to will be input into M gesture This shows that only when the time-series hand features are detected to be valid at the current moment t, can this moment become a gesture event. At the same time, because M gesrure For a multi-stage model, the gesture category feature vector can be obtained from the MobileViT module The dynamic gesture recognition module calling formula is as follows:

[0169]

[0170] Through strict input checking and confidence judgment, the accuracy and reliability of the output results of the dynamic gesture recognition module are guaranteed, enabling it to effectively process various types of hand posture data in actual application scenarios and reduce the occurrence of misjudgment.

[0171] In some examples, the feature dimension C obtained by the LSTM module after processing and upgrading the temporal hand features is u =128. C o =7, C o Represents the number of predefined gesture categories. In this example, there are seven common dynamic gestures, namely, swipe up, swipe down, swing left, swing right, click forward and backward, fist, and release. Set the threshold T g =0.65, used to measure the credibility of each element in the gesture category probability vector output by the MLP&Softmax module. g =768, C g is the dimension of the gesture category feature vector output by the MobileViT module. The 768-dimensional setting reflects the refinement of the MobileViT module in feature learning and representation.

[0172] In some examples, the loss function J of the dynamic gesture recognition model is gesture It is composed of CE loss. Assume that in the dataset M, the true category of the i-th sample is C i , loss function J gesture The calculation formula is:

[0173]

[0174] By traversing all samples, summing up the loss of each sample and taking the average, the performance of the model on the entire dataset is comprehensively evaluated. It is the gesture category probability vector predicted by the model for the i-th sample at time t. It means taking the probability value corresponding to the true category position from the predicted probability vector. The cross entropy loss is calculated for a single sample.

[0175] Since open source data cannot meet the training requirements of the dynamic gesture recognition model, a data collection method suitable for the training requirements is adopted. In some examples, during the training process of the dynamic gesture recognition model, dynamic gesture data is collected as a data set, including the steps of:

[0176] It is equipped with dual-mode cameras with multiple camera angles, and the optical center of the middle camera is facing the person being captured. Figure 7 The following is a simplified layout diagram of a dynamic gesture data collection device according to an embodiment of the present invention. As shown in the figure, in this example, five 1000w pixel dual-mode cameras camera1 to camera5 are configured, and each Meige camera selects a camera angle (-60°, -30°, 0°, 30°, 60°), and the optical center of the middle camera faces the person being collected.

[0177] The collector needs to make C in front of the screen at will. o The dual-mode camera captures the gesture video of each collector. In this example, a green curtain is selected as the background.

[0178] Steps S2 to S4 are executed to extract temporal hand features.

[0179] The characteristics of this data collection method are:

[0180] 1) The same gesture will have different backgrounds in different environments, which greatly interferes with the algorithm. Due to practical conditions, data collection cannot cover all scenes, so a green curtain background is used to facilitate background replacement using matting technology in the later stage.

[0181] 2) Gestures are presented differently under different imaging perspectives. Using multiple camera perspectives to obtain multi-perspective data can make up for problems such as finger occlusion that are prone to occur in a single perspective, enrich the model's cognition of the spatial morphology of gestures, and enhance its generalization ability.

[0182] 3) In different application scenarios, the imaging method and imaging quality are different. For example, in the car cabin, gesture recognition cameras mostly use infrared imaging, while in some conference applications, RGB imaging is mostly used. Therefore, high-resolution, IR / RGB dual-mode digital cameras are used to cover most application scenarios.

[0183] In the actual case, 70 volunteers (male:female ratio 6:4) were recruited to collect C o There are gesture videos, each of which has a duration ranging from 1s to 8s. These data constitute the dynamic gesture dataset of the present invention.

[0184] In some examples, during the training of the dynamic gesture recognition model, the Batch Size is set to 64, which means that the model will process 64 sample data at the same time in each training iteration. The learning rate decay strategy uses the cosine learning rate decrease, and the initial learning rate Lr gesture Set to 0.001, this value determines the step size of the model parameter update at the beginning of training. A total of 150 epochs are trained, which is set after comprehensive consideration of factors such as model complexity and data set size. The purpose is to allow the model to learn as many useful gesture recognition features as possible from the data without overfitting.

[0185] Furthermore, let the video FPS be 30, because the step size of the temporal filter in extracting temporal hand features is 1, and the time t is equivalent to the number of frames. As the video plays, that is, time passes, many or The gesture state sequence GS 3D and by or NULL composed of gesture classification feature sequence FS 3D However, a clear gesture command is required to implement subsequent operations (human-computer interaction, vehicle control, etc.). Therefore, in step S6, a time filter needs to be added again for filtering, and the filter width is W. G , which is different from the temporal filter in step S4 for extracting temporal hand features, the step size S G Equal to W G This means that step S6 can only output t+n*W G The gesture at the moment, and it must satisfy n∈N,t%W G ==0, this setting has the following two reasons:

[0186] Assume that the basic time length of the complete dynamic gesture action is W G , and there is no overlapping area between the gestures. G The filter with the width and step length can accurately capture a complete and independent gesture action;

[0187] Since steps S4 and S5 have fully considered and extracted the time-dependent feature relationship, the time filter in step S6 does not need to have an intersection with the sliding position of the previous step when sliding. This means that there is no need to repeatedly extract features that are closely related to adjacent moments, but rather a larger span can be used to focus on the complete gesture action, improving processing efficiency.

[0188] In some examples, step S6 includes:

[0189] S61, build time series, set width to W G The time filter window is used to capture the gesture category sequence and gesture classification feature sequence within a certain period of time;

[0190] At time t, let the time window slide to obtain a length of W G Gesture category sequence:

[0191]

[0192] in, for or t * in[t-(W G -1),t];

[0193] Get the gesture classification feature sequence at time t:

[0194]

[0195] in, for or NULL;

[0196] S62, preliminary screening, determining the number of gesture categories Is the percentage of sequence length less than T? valid , if so, then (background category), go to S68, if not, go to S63;

[0197] S63, statistical sorting, Sort the gesture categories by quantity to find the gesture category with the most occurrences and multiple gesture categories

[0198] S64, high confidence category judgment, judgment If so, then Go to S68, if not go to S65;

[0199] S65, medium confidence category judgment, judgment If yes, go to S67, if no, go to S66;

[0200] S66, low confidence category judgment, judgment If Enter S68;

[0201] S67, cosine similarity decision, based on gesture category And the corresponding gesture category feature vector set The feature vectors of the corresponding categories Calculate the average cosine similarity CS gestureij , the calculation formula is:

[0202]

[0203] Select CS gestureij The gesture category and category features corresponding to the minimum value are taken as the final output

[0204] Selection The feature with the smallest cosine similarity value is Do exponentially weighted updates,

[0205] It should be noted that after the preliminary screening in step S62, sometimes ambiguous gesture judgments may be obtained. This is because the time series characteristics of many gestures are similar, resulting in Based on the inspiration from face registration in face recognition algorithm, K-mean clustering algorithm is used to collect the most representative gesture category feature vectors from the validation set. i=0,1,...,C g -1, C g indivual As the registered feature vector of each category of dynamic gestures, in step S67, the cosine similarity comparison method is used to replace the quantity comparison method to further determine the final dynamic gesture.

[0206] S68, end.

[0207] In some examples, W G =66,W G As the width of the time filter window, it defines the range used for comprehensive judgment of gestures in the time dimension. valid =0.8, T valid It is used to measure the proportion of gesture categories in the sequence length. max_g =0.70, T max_g is the set threshold for high confidence category judgment. min_g=0.55, which is the threshold for low confidence category judgment. β = 0.9, β is a weight parameter in the exponential weighted update, which is set to maintain the stability of the feature vector.

[0208] The present invention also provides a dynamic gesture recognition system, which is applicable to the above-mentioned dynamic gesture recognition method. Figure 8 The structure diagram of a dynamic gesture recognition system according to an embodiment of the present invention is shown. As shown in the figure, the dynamic gesture recognition system 800 includes:

[0209] An acquisition unit 801 is configured to acquire a video stream;

[0210] A first extraction unit 802 is configured to extract a hand image based on a video stream;

[0211] A second extraction unit 803 is configured to extract hand spatial features based on the hand image;

[0212] The third extraction unit 804 is configured to extract temporal hand features based on the hand spatial features;

[0213] A gesture recognition unit 805 is configured to recognize the temporal hand features to obtain gesture categories and category features;

[0214] The decision unit 806 is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

[0215] Fig. 9 The system block diagram of a dynamic gesture recognition device according to an embodiment of the present invention is shown. As shown in the figure, the simulation device 900 may include an internal communication bus 901, a processor 902, a read-only memory (ROM) 903, a random access memory (RAM) 904, and a communication port 905. When applied on a personal computer, the simulation device 900 may also include a hard disk 906. The internal communication bus 901 may implement data communication between components of the simulation device 900. The processor 902 may make judgments and issue prompts. In some embodiments, the processor 902 may be composed of one or more processors. The communication port 905 may implement data communication between the simulation device 900 and the outside. In some embodiments, the simulation device 900 may send and receive information and data from the network through the communication port 905. The simulation device 900 may also include different forms of program storage units and data storage units, such as a hard disk 906, a read-only memory (ROM) 903, and a random access memory (RAM) 904, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 902. The processor 902 executes these instructions to implement the main part of the method. The result processed by the processor 902 is transmitted to the user device through the communication port 905 and displayed on the user interface.

[0216] The above-mentioned simulation method can be implemented as a computer program, stored in the hard disk 906, and loaded into the processor 902 for execution to implement the simulation method of the present application.

[0217] The present invention also provides a dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned dynamic gesture recognition methods when executing the computer program.

[0218] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned dynamic gesture recognition methods are implemented.

[0219] The specific implementation methods and technical effects of the dynamic gesture recognition device and the computer-readable storage medium can be found in the above-mentioned embodiments of the detection method provided by the present invention, which will not be described in detail here.

[0220] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0221] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0222] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0223] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0224] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention covers modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A dynamic gesture recognition method, comprising the steps of: S1, obtain video stream; S2, extracting a hand image based on the video stream; S3, extracting hand spatial features based on the hand image; S4, extracting temporal hand features based on the hand spatial features; S5, identifying the temporal hand features to obtain gesture categories and category features; S6, making a decision based on the gesture category and category features to determine a dynamic gesture.

2. The dynamic gesture recognition method according to claim 1, characterized in that: In step S2, the process of extracting the hand image includes the steps of: S21, normalize the image to unify the pixel value scale; S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes; S23, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region; S24, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

3. The dynamic gesture recognition method according to claim 1, characterized in that: In step S3, the process of extracting hand space features includes the steps of: S31, determine whether the hand image is valid, if not, proceed to S32, if valid, proceed to S33; S32, setting the hand space feature to an invalid value, outputting the hand space feature, and entering S34; S33, inputting the hand image into a 3D feature extraction model to extract hand spatial features, where the hand spatial features include 2D features and depth features; S34, end.

4. The dynamic gesture recognition method according to claim 3, characterized in that: In step 33, the 2D features of the hand image are obtained by using the 3D feature extraction model. and the deep feature Z t ; 2D features Convert to relative coordinate 2D features in image coordinate system The depth feature Z t Normalized to deep feature Z t_normal .

5. The dynamic gesture recognition method according to claim 3, characterized in that: After acquiring the hand spatial features in step S33, a feature filtering region is set, and features are further refined within the feature filtering region based on a spatial filtering method.

6. The dynamic gesture recognition method according to claim 5, characterized in that: Taking the current prediction position as the center, the R*R region is selected as the feature filtering region, and a Gaussian filter is used to further refine the features in the feature filtering region. The two-dimensional Gaussian distribution formula is: Where R = 3, σ = 1.5, (x, y) is the position in the image coordinate system, μ x , μ y is the current predicted position.

7. The dynamic gesture recognition method according to claim 1, characterized in that: Step S4 includes: Set the width to W T The time filter window is aligned with the rightmost side of the time filter to the current time t; Construct time series features. At time t, let the time window slide to obtain a length of W T Gesture time series features: The step size S of the temporal filter T is 1.

8. The dynamic gesture recognition method according to claim 1, characterized in that: Step S5 includes: S51, determining whether the hand space feature is valid, if not valid, proceeding to S52, if valid, proceeding to S53; S52, the gesture category is set to background, the category feature is set to an invalid value, the gesture category and category feature are output, and the process proceeds to S55; S53, inputting the hand space feature into a dynamic gesture recognition model to generate a gesture category, category feature and a gesture category probability vector; S54, determine the probability vector of gesture category The value in is the largest and greater than the set threshold T g , if so, select the gesture category probability vector The gesture category and category feature corresponding to the largest value in are output. If not, the gesture category is set to background, the category feature is set to invalid value, and the gesture category and category feature are output. S55, end.

9. The dynamic gesture recognition method according to claim 8, characterized in that: The dynamic gesture recognition model includes an LSTM module, a MobileViT module and an MLP&Softmax module. The LSTM module is used to extract and upgrade the input time series hand features to obtain The MobileViT module is used to convert the input Perform time and space fusion to obtain gesture categories and category features The MLP&Softmax module is used to transform the input gesture category and category features Get the gesture category probability vector 10. A dynamic gesture recognition system, applicable to the dynamic gesture recognition method according to claim 1, characterized in that: The dynamic gesture recognition system comprises: An acquisition unit, configured to acquire the video stream; A first extraction unit, configured to extract a hand image based on the video stream; a second extraction unit configured to extract hand spatial features based on the hand image; a third extraction unit, configured to extract temporal hand features based on the hand spatial features; a gesture recognition unit configured to recognize the temporal hand features to obtain gesture categories and category features; The decision unit is configured to make a decision based on the gesture category and category features to determine the dynamic gesture.

11. A dynamic gesture recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the dynamic gesture recognition method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic gesture recognition method according to any one of claims 1 to 9 are implemented.

Images