Dynamic gesture recognition method based on ResNet and ResNeXt

Abstract

The invention discloses a dynamic gesture recognition method based on ResNet and ResNeXt, and relates to the technical field of man-machine interaction, and the method comprises the steps that a detector detects incoming video data, if it is detected that action gestures exist in the video data, a classifier is activated, and each gesture is activated at a single time; and the classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures. Accurate recognition and quick response to dynamic gestures are achieved through ResNet and ResNeXt, single activation is adopted when the classifier is activated after the detector detects the gesture action, the duration of the gesture is regardless of, the misjudgment rate of the detector is reduced, and the accuracy of dynamic gesture recognition is further improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a dynamic gesture recognition method based on ResNet and ResNeXt. Background Art

[0002] With the continuous advancement of science and technology, gesture interaction technology has been widely integrated into daily life in this embodiment. Among them, dynamic gesture recognition, as a key link in gesture interaction, is attracting more and more research attention. However, the recognition of dynamic gestures faces many challenges, including the complexity brought by the changes in gesture movements in space, and the high requirements of gesture recognition for real-time performance. Therefore, how to accurately and quickly recognize dynamic gestures has become an important issue.

[0003] Related research has also been carried out in traditional technologies. Chinese patent CN118279992A is a gesture recognition method and system based on LSTM training model. Through LSTM training model, grid data of gestures to be recognized are obtained frame by frame. Chinese patent CN101882000A is a gesture recognition method based on acceleration sensor. Acceleration sensor is studied to extract rotation feature codes of all samples written by users in three-dimensional space.

[0004] However, most previous studies have only considered acceptable classification accuracy and attempted to improve offline classification accuracy in gesture recognition, while ignoring fast reaction time, resource efficiency, single activation for each action, etc. Summary of the invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a dynamic gesture recognition method based on ResNet and ResNeXt, comprising:

[0007] The detector detects the incoming video data and activates the classifier if an action gesture is detected in the video data;

[0008] The classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures.

[0009] In a possible implementation, the detector detects the incoming video data, and activates the classifier if an action gesture is detected in the video data, including:

[0010] The detector introduces residual blocks and identity connections to ResNet, allowing the input to directly skip the output of some layers and then be added to the output of these layers; in ResNet, a residual block can be expressed as: y = F(x, W i )+xy=F(x,W i )+x;

[0011] Where: x is the input of the residual block, y is the output of the residual block, and F(x,Wi) represents the residual mapping learned by the layer with weight Wi;

[0012] In the forward propagation process of the residual block, the input x passes through one or more convolutional layers to obtain the residual map F(x,Wi);

[0013] Then, the residual map is added to the input x and the output y is obtained through a nonlinear activation function: y = ReLU(F(x,W i )+x).

[0014] In one possible implementation, the detector is trained using weighted cross entropy loss.

[0015]

[0016] Recall>0.97

[0017] Where: C no-gesture For the non-gesture class, w no-gesture is the selected classification weight; C gesture is the gesture class, and Recall is the recall rate.

[0018] In one possible implementation, the detector output probabilities are post-processed and a counter of consecutive no-gesture predictions is set in the decision to deactivate the classifier:

[0019]

[0020] Among them, δ is an indicator function, when C i =C no-gesture It is 1 when n is the number of consecutive predictions, otherwise it is 0.

[0021] In one possible implementation, the probability output by the detector is post-processed, including: adding the original softmax probability predicted by the detector to the queue Q k , of size k; these raw values ​​are filtered to obtain the final detector decision, eliminating most of the misclassifications in the continuous predictions.

[0022] In one possible implementation, the raw softmax probability predicted by the detector is added to the queue Q k , of size k; filtering these raw values ​​to obtain the final detector decision eliminates most of the misclassifications in the continuous predictions, including:

[0023] Let P t is the predicted probability of the detector at time t, C t is the true classification at time t, Q k is a queue of size k, which stores the previous k prediction probabilities P t-k+1 , P t-k+1 , ..., P t , queue Q k It is expressed as:

[0024] Q k =[P t-k+1 ,P t-k+2 ,...,P t ]

[0025] If the detector's prediction at time t-1 is correct: C t-1 = argmax(P t-1 );

[0026] However, if the classification at time t is wrong: C t ≠argmax(P t );

[0027] The indicator function Performance_Decreas is used for description:

[0028]

[0029] Queue Q k The expression of each element in after average filtering is:

[0030]

[0031] The expression after exponential weighted average filtering is:

[0032]

[0033] The expression after median filtering is:

[0034] MedianFilter(Q k )=Median([P t-k+1 ,P t-k+2 ,...,P t ])

[0035] The specific expression of the probability distribution after filtering is as follows:

[0036]

[0037] In one possible implementation, the classifier uses CNN and ResNext modules, wherein the Bottleneck structure in the ResNext module consists of three convolutional layers:

[0038] The first one is the first 1×1 convolutional layer used to reduce the number of channels. The number of input channels is C in , the number of output channels is 4C in , to reduce the computational effort of the subsequent 3×3 convolutional layer;

[0039] The second 3×3 convolutional layer is used to extract features. The number of input channels of this layer is 4C in , the number of output channels is also 4C in , extract high-level features of input features through 3x3 convolution;

[0040] The third 1×1 convolutional layer is used to restore the number of channels. The number of input channels of this layer is 4C in , the number of output channels is C out , restore the number of channels of the feature to a value that matches the input feature or subsequent layer through 1×1 convolution;

[0041] The Bottleneck structure output is: in, is the first 1×1 convolutional layer; GroupConv 3×3 is the second 3×3 convolutional layer; It is the third 1×1 convolutional layer.

[0042] In a possible implementation, the classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures, including:

[0043] Calculate the number of convolution kernels in each group and the number of input channels processed by each group of convolution kernels:

[0044]

[0045] Perform independent convolution operations on each group: for i=1, 2, ..., G, perform the following operations:

[0046]

[0047] Among them, Conv k×k Represents a convolution operation using a k×k convolution kernel

[0048] Concatenate the convolution outputs of each group to form the final output feature map:

[0049] output=Concat(output group1 ,output group2 ,...,output groupG );

[0050] Where: C in is the number of channels of input feature data, C out The number of channels for output feature data, k×k is the size of the convolution kernel, G is the number of groups for grouped convolution, and N is kernels_per_group is the number of convolution kernels in each group, N channels_per_group_in is the number of input channels processed by each set of convolution kernels, is the convolution input of the i-th group, is the convolution output of the i-th group, Conv k×k Represents a convolution operation using a k×k convolution kernel.

[0051] In one possible implementation, the detector activates the classifier using single activation to ensure that each gesture is activated only once. The single activation sigmoid-like weight function is formulated as:

[0052]

[0053] Through the weighted average strategy, the single activator can make a decision in the middle and late stages of the gesture after capturing the core part of the gesture.

[0054] In the embodiments of the present application, accurate recognition and rapid response of dynamic gestures are achieved by utilizing ResNet and ResNeXt, and a single activation is adopted when activating the classifier after the detector detects a gesture action, regardless of the duration of the gesture, thereby reducing the misjudgment rate of the detector and further improving the accuracy of dynamic gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a dynamic gesture recognition method based on ResNet and ResNeXt provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the system workflow provided by the embodiment of the present application;

[0057] Figure 3 A schematic diagram of dynamic gesture recognition performance analysis provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of a ResNet block used in the detector architecture provided in an embodiment of the present application;

[0059] Figure 5 Schematic diagram of original (top) and weighted (bottom) classification scores provided for embodiments of the present application;

[0060] Figure 6 The category score weight function image provided by the embodiment of the present application;

[0061] Figure 7 Schematic diagram of the ResNeXt block used in the classifier architecture provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present solution is described below in conjunction with the accompanying drawings and specific implementation methods.

[0063] See also Figure 1 The dynamic gesture recognition method based on ResNet and ResNeXt provided in this embodiment includes:

[0064] S101, the detector detects the incoming video data, and activates the classifier if an action gesture is detected in the video data, wherein each gesture is activated once.

[0065] In the incoming data, the sliding window starts at the beginning of the detector queue in the classifier queue; if the detector recognizes the action / gesture, the classifier is activated. Figure 2 The detector and classifier are provided by sequences of sizes n and m (n is much smaller than m and contains overlapping factors), respectively, and both use the same stride value s. Although a higher s value can reduce resource usage, in order not to miss any gestures and ensure better performance, s is selected as 1 in this embodiment.

[0066] The purpose of the detector is to distinguish between the gesture class and the no-gesture class. Its main and only role is to act as a switch for the classifier model, meaning that if it detects a gesture, then the classifier is activated and fed with frames in the classifier queue.

[0067] See also Figure 3 , shows the probability score of a detector being activated at the beginning of a gesture and remaining activated until the gesture ends. The second figure shows the classification scores for each class in different colors. The third figure applies a weighted average filter to the raw classification scores, removing ambiguity between possible gesture candidates. The figure below illustrates a single activation, with red arrows indicating early detections of the action and black arrows indicating detections after the gesture is complete.

[0068] exist Figure 3In the figure, the top graph shows the probability scores of detectors that are active when a gesture is performed, while the detectors remain inactive during the rest of the time. The classifier is activated only when the detector successfully detects a gesture. This is critical because in real-time gesture recognition applications, most of the time no gestures are being performed. Therefore, there is no need to keep the high-performance classifier active all the time, which can significantly reduce the system's memory usage and power consumption.

[0069] The following figure presents the raw classification scores of each category by different colors. From this chart, we can observe that in some cases, similar category scores rise at the same time, especially at the beginning of the gesture. This phenomenon increases the complexity of classification because the system needs to accurately distinguish these similar categories.

[0070] To resolve this ambiguity, we weighted the class scores to avoid making decisions too early at the beginning of a gesture. Figure 3 This is reflected in the third chart.

[0071] In addition, the following chart shows the single activation, where the red arrow indicates early detection and the black arrow indicates detection after the gesture is completed. To evaluate the performance of the system, we used the Levenshtein distance as the evaluation metric, which can measure misclassification, multiple detections, and missed detections at the same time, and has high applicability and evaluation value.

[0072] In the detector, ResNet is used in this embodiment. ResNet introduces residual blocks and identity connections, allowing the input to directly skip the output of some layers and then be added to the output of these layers. This can effectively alleviate the gradient vanishing problem, simplify the network design, and improve the model performance.

[0073] In ResNet, a residual block can be expressed as:

[0074] y=F(x,W i )+xy=F(x,W i )+x

[0075] Among them, x is the input of the residual block, y is the output of the residual block, F(x,W i ) indicates that the weight is W i The residual mapping learned by the layer .

[0076] In the forward propagation process of the residual block, the input x passes through one or more convolutional layers (or other types of layers) to obtain the residual map F(x,W i). Then, this residual map is added to the input x and passed through a nonlinear activation function (such as ReLU) to get the output y. This process can be expressed as:

[0077] y=ReLU(F(x,W i )+x)

[0078] Since the overall accuracy of the system is highly dependent on the performance of the detector, the detector in this embodiment is required to be (a) robust, (b) accurately detect true positives (gestures), and (c) lightweight since it runs continuously.

[0079] For (a), let the detector run on fewer frames than the classifier.

[0080] For (b), place the detector queue at the very beginning of the classifier queue, such as Figure 2 This allows the detector to activate the classifier whenever a gesture starts, regardless of the gesture duration. In addition, to reduce the possibility of false positives (i.e., to obtain a higher recall rate), the detector model is trained using weighted cross entropy loss. For the no gesture class (denoted as C no-gesture , the selected classification weight is w no-gesture ) and gesture class (denoted as C gesture , the selected classification weight is w no-gesture ).

[0081]

[0082] Recall>0.97

[0083] The classification weights selected in this embodiment are 1 and 3 respectively, because the experiments in this embodiment show that this ratio is sufficient to have a recall rate of 97+% in the ChaLearn Gesture Challenge dataset.

[0084] In addition, in this embodiment, the output probability is post-processed, and a counter of the number of consecutive no-gesture predictions is set in the decision to deactivate the classifier:

[0085]

[0086] Among them, δ is an indicator function, when C i =C no-gesture It is 1 when n is the number of consecutive predictions, otherwise it is 0.

[0087] For (c), use Figure 4The ResNet blocks in the ResNet-18 architecture are used to construct the feature size of each layer, as shown in Table 1. This results in a parameter size of less than 1M (≈862K). F and N are the number of feature channels and the number of blocks in the corresponding layer, respectively. Figure 3 The BN, ReLU, and group in it are batch normalization, rectified linear unit nonlinearity, and group convolution number, respectively.

[0088] Table 1 Detector (ResNet-18) and classifier (ResNeXt-101) architecture

[0089]

[0090] In dynamic gestures, even if the previous prediction of the detector is correct, any misclassification will degrade the overall performance of the proposed architecture.

[0091] In order to utilize previous predictions, this embodiment adds the raw softmax probabilities predicted by the previous detector to the queue Q k , size is k, and these raw values ​​are filtered to obtain the final detector decision. Using this method, the detector improves the confidence of the decision and eliminates most of the misclassifications in the continuous prediction. In this embodiment, P t is the predicted probability of the detector at time t (probability distribution after softmax), C t is the true classification at time t, Q k is a queue of size k, which stores the previous k prediction probabilities P t-k+1 , P t-k+1 , ..., P t .

[0092] If the detector's prediction at time t-1 is correct: C t-1 = argmax(P t-1 );

[0093] However, if the classification at time t is wrong: C t ≠argmax(P t );

[0094] Then, the overall system performance will be reduced. In order to clearly reflect this situation, an indicator function can be used to describe it in this embodiment:

[0095]

[0096] In the experimental setting of this embodiment, the size of the queue (k) is selected as 4. Moreover, through experimental verification, when the parameter s is set to 1, the system achieves the best performance.

[0097] In this embodiment, for queue Q k Each element in is processed by averaging, exponential weighted averaging, and median filtering. k It can be expressed as:

[0098] Q k =[P t-k+1 ,P t-k+2 ,...,P t ]

[0099] After average filtering, the expression is:

[0100]

[0101] The expression after exponential weighted average filtering is:

[0102]

[0103] The expression after median filtering is:

[0104] MedianFilter(Q k )=Median([P t-k+1 ,P t-k+2 ,...,P t ])

[0105] After calculation and analysis, the final decision is made in this embodiment based on the probability distribution after filtering. The specific expression is as follows:

[0106]

[0107] It is worth noting that even if the prediction P at time t t is wrong, but the result after using median filtering still correctly points to category 1. This phenomenon fully demonstrates the remarkable effect of median filtering in correcting misclassification in continuous prediction. For this reason, among the three filtering strategies of average filtering, exponential weighted average filtering and median filtering, median filtering is selected as the optimal strategy in this embodiment.

[0108] In real-time gesture recognition systems, it is very important to shorten the reaction time and single activation of each gesture. Pavlovic et al. pointed out that dynamic gestures have preparation, nucleus (peak) and retraction parts. Among all the parts, the nucleus is the most discriminative one, because in this embodiment, it can be determined which gestures are performed in the nucleus part and even predicted before it ends.

[0109] One-shot activation is achieved through a two-level control mechanism. Either a gesture is detected when the confidence measure reaches a threshold level before the gesture actually ends (early detection), or a gesture is predicted when the detector invalidates the classifier (late detection). In late detection, it is assumed in this embodiment that the detector should not miss any gestures, since the detector is guaranteed to have a very high recall in this embodiment.

[0110] The most critical part of early detection is that the gesture should be detected after its core part to obtain better recognition performance. Because some gestures may contain similar preparation parts, which creates an ambiguity at the beginning of the gesture, such as Figure 5 as shown in the top figure.

[0111] In the above graph, this embodiment observes that there is a lot of noise at the beginning of all gestures; however, the classifier becomes more confident towards the end of each gesture. The following graph shows that this embodiment can eliminate this noisy part by assigning a smaller weight to the beginning of the gesture.

[0112] Therefore, in this embodiment, the weighted average method is applied to the category scores, and its weight function image is as follows: Figure 6 As shown:

[0113] According to the image, the formula for the sigmoid-like weight function for a single activation is:

[0114]

[0115] Through this weighted average strategy, the single activator in this embodiment is forced to make a decision in the middle and late stages of the gesture after capturing the core part of the gesture. On the other hand, due to the variation in gesture duration, a confidence measurement is required in this embodiment for real-time early detection. Therefore, in this embodiment, it is decided to use the difference in the weighted average scores of each category as the confidence indicator for early detection in this embodiment. When the detector turns on the classifier, the weighted average probability of each class is calculated in each iteration. If the difference between the two highest average probabilities is greater than a threshold t_early, then early detection is triggered. Otherwise, this embodiment waits for the detector to turn off the classifier, at which time the category with the highest score greater than t_late (fixed to 0.15 because it showed the best results in the experiments in this embodiment) is predicted as a delayed detection.

[0116] Unlike offline testing, which usually only considers the accuracy of the class, this embodiment must also consider the following scenarios for real-time evaluation:

[0117] (1) Misclassification of gestures by the classifier

[0118] (2) The detector does not detect the gesture

[0119] (3) Multiple detections in one gesture

[0120] Taking these into account, this embodiment uses Levenshtein distance as an evaluation metric for online experiments. Levenshtein distance is a metric that measures the distance between sequences by calculating the number of item-level changes (insertions, deletions, or substitutions) that transform one sequence into another. In the experiment in this embodiment, the same gesture is performed twice, corresponding to a sequence and an item in the sequence, respectively.

[0121] For example, Ground Truth and predicted gestures:

[0122] GroundTruth[1,2,3,4,5,6,7,8,9]

[0123] Predicted[1,2,7,4,5,6,6,7,8,9]

[0124] In this experiment, the Levenshtein distance is 2: one of the two detected "6"s is deleted, and "7" is replaced by "3". This distance is averaged by the number of true target categories in this embodiment. For this case, the average distance is 2 / 9 = 0.2222, and then this value is subtracted from 1 in this embodiment, which is called Levenshtei, because this embodiment wants to measure the closeness of the results in this embodiment (in this work, it is n precision), which is equal to (1-0.2222) × 100 = 77.78%.

[0125] S102, the classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures.

[0126] Since there is no restriction on the size and complexity of the model in this embodiment, any architecture that provides good classification performance can be selected as the classifier. This leads to the use of CNN and ResNext-101 as the classifier models in this embodiment. For ResNeXt-101, the selected model parameters are shown in Table 1. The ResNeXt block is as follows: Figure 7 shown.

[0127] Here, the following parameters are defined in this embodiment to clarify the working principle of the ResNeXt block:

[0128] Assume: C in is the number of channels of input feature data, C out The number of channels for output feature data, k×k is the size of the convolution kernel, and G is the number of groups for grouped convolution. kernels_per_group is the number of convolution kernels in each group, N channels_per_group_inis the number of input channels processed by each set of convolution kernels, is the convolution input of the i-th group, is the convolution output of the i-th group.

[0129] Calculate the number of convolution kernels in each group and the number of input channels processed by each group of convolution kernels:

[0130]

[0131] Perform a separate convolution operation on each group:

[0132] For i=1, 2, ..., G, the following operations are performed:

[0133]

[0134] Among them, Conv k×k Represents a convolution operation using a k×k convolution kernel.

[0135] Concatenate the convolution outputs of each group to form the final output feature map:

[0136] output=Concat(output group1 ,output group2 ,...,output groupG )

[0137] Bottleneck structure

[0138] The Bottleneck structure in the ResNext architecture of the present invention consists of three convolutional layers:

[0139] The first 1×1 convolutional layer: used to reduce the number of channels. The number of input channels is C in , the number of output channels is 4C in , to reduce the computational effort of the subsequent 3×3 convolutional layers.

[0140] The second 3×3 convolutional layer: used to extract features. The number of input channels of this layer is 4C in , the number of output channels is also 4C in This step is the core of the Bottleneck structure, which extracts high-level features of the input features through 3x3 convolution.

[0141] The third 1×1 convolutional layer: used to restore the number of channels. The number of input channels of this layer is 4C in , the number of output channels is C out This step restores the number of channels of the feature to a value that matches the input feature or subsequent layer through a 1×1 convolution.

[0142]

[0143] in, is the first 1×1 convolutional layer; GroupConv 3×3 is the second 3×3 convolutional layer; It is the third 1×1 convolutional layer.

[0144] To avoid the problem of gradient disappearance or gradient explosion, a direct connection (or "skip connection") is introduced to pass the input feature x to the subsequent layer, so that the network can more easily learn the identity mapping during training.

[0145] The formula of residual connection can be expressed as:

[0146] y=x+output=Concat(output group1 ,output group1 ,...,output groupG )

[0147] In order to prevent overfitting, the present invention requires more training data. Therefore, in this embodiment, the classifier architecture is first pre-trained on the Jester data set, and then the model in this embodiment is fine-tuned on the ChaLearn Gesture Challenge data set. This method improves the accuracy of training and greatly shortens the training time.

[0148] In this embodiment, the unique advantages of special variants of ResNet [a lightweight, shallow CNN gesture detection (detector)] and ResNeXt [an offline trained deep CNN gesture classification (classifier)] are utilized to achieve accurate recognition and rapid response of gestures.

[0149] In the detector part, let the detector run on fewer frames than the classifier and put the detector queue at the beginning of the classifier queue, which allows the detector to activate the classifier whenever a gesture starts, regardless of the gesture duration. In addition, in order to reduce the possibility of false positives (i.e., to obtain a higher recall rate), the detector model is trained with weighted cross entropy loss, with a recall rate of 97+% in the ChaLearn Gesture Challenge dataset. ResNet blocks construct the ResNet-18 architecture, and the feature size of each layer is very small.

[0150] In the classifier part, CNN and ResNext-101 are used as our classifier models. The classifier architecture is pre-trained on the Jester dataset, and then our model is fine-tuned on the ChaLearn Gesture Challenge dataset. This method improves the accuracy of training and greatly shortens the training time.

[0151] In the post-processing part, since in dynamic gestures, even if the previous prediction of the detector is correct, any wrong classification will degrade the overall performance of the proposed architecture. We add the raw softmax probabilities predicted by the previous detector to the queue Q k , size is k, multiple filters are tested, and finally the best median filter is used to filter these raw values ​​to obtain the final detector decision. Using this method, the detector improves the confidence of the decision and eliminates most of the misclassifications in the continuous prediction.

[0152] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0153] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A dynamic gesture recognition method based on ResNet and ResNeXt, characterized in that: include: The detector detects the incoming video data, and activates the classifier if an action gesture is detected in the video data, wherein each gesture is activated once; The classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures.

2. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 1, characterized in that: The detector detects the incoming video data, and activates the classifier if an action gesture is detected in the video data, including: The detector introduces residual blocks and identity connections to ResNet, allowing the input to directly skip the output of some layers and then be added to the output of these layers; in ResNet, a residual block can be expressed as: y = F(x, W i )+xy=F(x,W i )+x; Where: x is the input of the residual block, y is the output of the residual block, and F(x,Wi) represents the residual mapping learned by the layer with weight Wi; In the forward propagation process of the residual block, the input x passes through one or more convolutional layers to obtain the residual map F(x,Wi); Then, the residual map is added to the input x and the output y is obtained through a nonlinear activation function: y = ReLU(F(x,W i )+x).

3. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 2, characterized in that: The detector is trained using weighted cross entropy loss. Recall>0.97 Where: C no-gesture For the non-gesture class, w no-gesture is the selected classification weight; C gesture is the gesture class, and Recall is the recall rate.

4. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 3, characterized in that: The detector output probabilities are post-processed and a counter of consecutive no-gesture predictions is set in the decision to deactivate the classifier: Among them, δ is an indicator function, when C i =C no-gesture It is 1 when n is the number of consecutive predictions, otherwise it is 0.

5. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 4, characterized in that: The probability of the detector output is post-processed, including: adding the original softmax probability predicted by the detector to the queue Q k , of size k; these raw values ​​are filtered to obtain the final detector decision, eliminating most of the misclassifications in the continuous predictions.

6. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 5, characterized in that: The raw softmax probability predicted by the detector is added to the queue Q k , size is k; These raw values ​​are filtered to obtain the final detector decision, eliminating most of the misclassifications in the continuous predictions, including: Let P t is the predicted probability of the detector at time t, C t is the true classification at time t, Q k is a queue of size k, which stores the previous k prediction probabilities P t-k+1 , P t-k+1 , ..., P t , queue Q k It is expressed as: Q k =[P t-k+1 ,P t-k+2 ,...,P t ] If the detector's prediction at time t-1 is correct: C t-1 = argmax(P t-1 ); However, if the classification at time t is wrong: C t ≠argmax(P t ); The indicator function Performance_Decreas is used for description: Queue Q k The expression of each element in after average filtering is: The expression after exponential weighted average filtering is: The expression after median filtering is: MedianFilter(Q k )=Median([P t-k+1 ,P t-k+2 ,...,P t ]) The specific expression of the probability distribution after filtering is as follows:

7. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 1, characterized in that: The classifier uses CNN and ResNext modules, where the Bottleneck structure in the ResNext module consists of three convolutional layers: The first 1×1 convolutional layer is used to reduce the number of channels. The number of input channels is C in , the number of output channels is 4C in , to reduce the computational effort of the subsequent 3×3 convolutional layer; The second 3×3 convolutional layer is used to extract features. The number of input channels of this layer is 4C in , the number of output channels is also 4C in , extract high-level features of input features through 3x3 convolution; The third 1×1 convolutional layer is used to restore the number of channels. The number of input channels of this layer is 4C in , the number of output channels is C out , restore the number of channels of the feature to a value that matches the input feature or subsequent layer through 1×1 convolution; The Bottleneck structure output is: in, is the first 1×1 convolutional layer; GroupConv 3×3 is the second 3×3 convolutional layer; It is the third 1×1 convolutional layer.

8. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 7, characterized in that: The classifier classifies the video frames corresponding to the action gestures detected by the detector to identify different dynamic gestures, including: Calculate the number of convolution kernels in each group and the number of input channels processed by each group of convolution kernels: Perform independent convolution operations on each group: for i=1, 2, ..., G, perform the following operations: Among them, Conv k×k Represents a convolution operation using a k×k convolution kernel The convolution outputs of each group are concatenated to form the final output feature map: output=Concat(output group1 ,output group2 ,...,output groupG ); Where: C in is the number of channels of input feature data, C out The number of channels for output feature data, k×k is the size of the convolution kernel, G is the number of groups for grouped convolution, and N is kernels_per_group is the number of convolution kernels in each group, N channels_per_group_in is the number of input channels processed by each set of convolution kernels, is the convolution input of the i-th group, is the convolution output of the i-th group, Conv k×k Represents a convolution operation using a k×k convolution kernel.

9. The dynamic gesture recognition method based on ResNet and ResNeXt according to claim 1, characterized in that: The detector activation classifier uses single activation to ensure that each gesture is activated only once. The formula of the single activation sigmoid weight function is: Through the weighted average strategy, the single activator can make a decision in the middle and late stages of the gesture after capturing the core part of the gesture.

Images