Dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution

Abstract

The application provides a dynamic gesture recognition method based on light-weight multivariate space-time convolution, which comprises the following steps: constructing a space feature extraction module based on a pseudo 3D gated attention fusion network; extracting multi-scale space features by using the space feature extraction module; injecting a guided heat map by using a gated attention fusion module to enhance key region features and suppress background interference, so as to obtain a space refinement feature sequence; decomposing the space refinement feature sequence into multiple sub-variables; capturing long-range and local time dependence in parallel by using a modern convolution module; and modeling the intra-variable and inter-variable relationships respectively by using a decoupling feature interaction network, so as to obtain a dynamic gesture recognition result. By combining the multivariate feature decomposition strategy with the modern convolution, the double-branch design advantage of the modern convolution is to balance long-range dependence and local details, solve the limitations of traditional convolution in capturing long-range time dependence, and model complex dynamics.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution. Background Technology

[0002] Dynamic gesture recognition is a core technology in the field of human-computer interaction, with wide and important applications in virtual reality, smart cockpits, and sign language translation. However, different scenarios have different requirements. For example, sign language translation requires lightweight, low-latency models, while gesture recognition and interaction in smart cockpits require models with high accuracy and robustness.

[0003] Most existing advanced dynamic gesture recognition models employ Transformer models or hybrid models. These models suffer from the following problems: First, their computational complexity exhibits a doubly increasing trend. This introduces unacceptable latency in real-time demanding scenarios, such as sign language translation and gesture interaction in games, limiting their applicability in real-time environments. Simultaneously, they impose extremely high energy consumption on the corresponding devices. For example, in real-world VR headsets, high latency and high energy consumption significantly degrade the user experience. Furthermore, besides computational costs, these models often tend to stack a large number of parameter-intensive encoder blocks, resulting in a "bloated architecture" with model parameters often reaching tens or even hundreds of millions. This poses a significant challenge to the deployment of edge devices, which have severely limited memory and storage, creating a sharp contradiction with the growing edge intelligence requirements in applications such as VR headsets and smart cockpits. Summary of the Invention

[0004] In view of the above situation, the main objective of this invention is to propose a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution to solve the above-mentioned technical problems.

[0005] This invention proposes a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution. The construction of the model includes the following steps:

[0006] Step 1: Construct a spatial feature extraction module based on a pseudo-3D gated attention fusion network, and a temporal modeling module based on a multivariate decomposition temporal convolutional network. The spatial feature extraction module and the temporal modeling module together form a lightweight multivariate spatiotemporal convolutional model. The pseudo-3D gated attention fusion network includes: a basic feature extraction layer, an Inception Block module, and a gated fusion attention module. The multivariate decomposition temporal convolutional network includes: a variable decomposition unit, a modern convolutional module, a decoupled feature interaction network, and a variable aggregation unit.

[0007] Step 2: Obtain dynamic gesture video clips, and use the spatial feature extraction module to sequentially perform multi-scale spatial feature extraction and prior enhancement processing on the dynamic gesture video clips to obtain a spatially refined feature sequence.

[0008] Step 3: Use the time modeling module to perform feature decomposition, multi-scale time dependency capture, and decoupled feature interaction processing on the spatially refined feature sequence in sequence to obtain the time-varying feature sequence;

[0009] Step 4: Process the time-varying feature sequence sequentially through average pooling, layer normalization, and fully connected layers to obtain the prediction score matrix;

[0010] Step 5: Construct the cross-entropy loss function based on the prediction score matrix, and use the cross-entropy loss function to train the lightweight multivariate spatiotemporal convolutional model to obtain the trained lightweight multivariate spatiotemporal convolutional model; input the dynamic gesture video clip into the trained lightweight multivariate spatiotemporal convolutional model to obtain the dynamic gesture recognition result.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] 1. Reduce the computational cost of network inference. Existing dynamic gesture recognition models suffer from excessive inference costs due to the introduction of multi-head attention mechanisms, resulting in suboptimal performance in scenarios requiring low latency and fast response. This invention presents a lightweight multivariate spatiotemporal convolutional model based on convolution technology, containing only 9.6 million parameters. Compared to existing Transformer models and their hybrid models with tens or hundreds of millions of parameters, the DSTCNet model is significantly more lightweight. In terms of performance, the DSTCNet model also performs admirably. While maintaining its lightweight design, its performance on multiple public datasets is close to that of existing models, and it even outperforms existing models on some datasets.

[0013] 2. Achieving an efficient and accurate spatial feature extraction network. Efficient spatial representation is crucial for gesture recognition. While 3D convolutional neural networks can preserve spatiotemporal information, they are computationally expensive; 2D convolutional networks are efficient, but their ability to represent temporal information is poor. This invention addresses these shortcomings by proposing an efficient and accurate alternative: using pseudo-3D convolution to decompose the 3D kernel into spatial and temporal components. Furthermore, considering the complexity of gesture recognition in the real world, prior knowledge (such as hand joint heatmaps) is injected, and spatial priors are used as input, fused end-to-end with visual features. This allows for easy adaptation to different types of prior information, resulting in a simple and generalizable spatial feature extraction method.

[0014] 3. A novel temporal modeling module is proposed. Traditional convolutional methods, due to their limited receptive field, cannot connect keyframe information with long intervals in temporal modeling, leading to errors in judging events that require contextual concatenation. Transformer-based methods also capture the context of long sequences with quadratic complexity. The temporal modeling module proposed in this invention is a pure convolutional method with high expressive power for time series. By combining a multivariate feature decomposition strategy (reducing feature entanglement) with modern convolution, the dual-branch design of modern convolution has the advantage of balancing long-range dependencies and local details, overcoming the limitations of traditional convolution in capturing long-range temporal dependencies and modeling complex dynamics.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution proposed in this invention.

[0017] Figure 2 This is a diagram illustrating the overall framework of a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution proposed in this invention.

[0018] Figure 3 This is a structural diagram of the Inception Block module proposed in this invention;

[0019] Figure 4 This is a structural diagram of the gated fusion attention module proposed in this invention;

[0020] Figure 5 This is a schematic diagram of the decoupled feature interaction network proposed in this invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0022] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0023] Please see Figure 1This embodiment provides a dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution, the method including the following steps:

[0024] Step 1: Construct a spatial feature extraction module based on a pseudo-3D gated attention fusion network, and a temporal modeling module based on a multivariate decomposition temporal convolutional network. Use the spatial feature extraction module and the temporal modeling module to form a lightweight multivariate spatiotemporal convolutional model. The pseudo-3D gated attention fusion network includes: a basic feature extraction layer, an Inception Block module, and a gated fusion attention module. The multivariate decomposition temporal convolutional network includes: a variable decomposition unit, a modern convolutional module, a decoupled feature interaction network, and a variable aggregation unit.

[0025] Step 2: Obtain dynamic gesture video clips, and use the spatial feature extraction module to sequentially perform multi-scale spatial feature extraction and prior enhancement processing on the dynamic gesture video clips to obtain a spatially refined feature sequence.

[0026] Please see Figure 2 and Figure 3 In step 2, the spatial feature extraction module is used to sequentially perform multi-scale spatial feature extraction and prior enhancement processing on the dynamic gesture video clips to obtain a spatially refined feature sequence. This includes the following sub-steps:

[0027] A preliminary feature map is obtained by using a basic feature extraction layer to perform preliminary feature encoding on dynamic gesture video clips;

[0028] The Inception Block module is used to perform multi-scale feature fusion on the preliminary feature map to obtain a multi-scale fused feature map. The Inception Block module includes four parallel convolutional branches.

[0029] Guided heatmaps are obtained from dynamic gesture video clips. The guided heatmaps are then fused with multi-scale fusion feature maps using a gated fusion attention module to obtain spatially refined feature sequences.

[0030] Furthermore, the step of using the Inception Block module to perform multi-scale feature fusion on the preliminary feature map to obtain a multi-scale fused feature map specifically includes the following sub-steps:

[0031] The initial feature maps are assigned to four parallel convolutional branches in the Inception Block module for processing to obtain the first branch features, the second branch features, the third branch features, and the fourth branch features;

[0032] The first convolutional branch is a 1×1×1 convolutional layer, the second convolutional branch includes a 1×1×1 convolutional layer and a 1×3×3 convolutional layer in sequence, the third convolutional branch includes a 1×1×1 convolutional layer and a 1×3×3 convolutional layer in sequence, and the fourth convolutional branch includes a max pooling layer and a 1×1×1 convolutional layer in sequence.

[0033] The features from the first branch, the second branch, the third branch, and the fourth branch are concatenated to obtain a multi-scale fused feature map.

[0034] Furthermore, the step of obtaining the guiding heatmap based on dynamic gesture video clips specifically includes the following sub-steps:

[0035] A two-dimensional coordinate grid is constructed based on video frames of dynamic gesture video clips. The response values ​​of hand key points detected in the video frames in the two-dimensional coordinate grid are calculated using a two-dimensional Gaussian kernel function. The following relationship exists in the correspondence process:

[0036] ;

[0037] in, Indicates the first [number]th ... The first detected in the frame Key hand features are represented in a two-dimensional coordinate grid. The response value at that location, Indicates the index of the video frame. Indexes representing key points of the hand. Represents pixel coordinates in a two-dimensional coordinate grid. Represents an exponential function. Indicates the first [number]th ... The first detected in the frame Coordinates of key points on the hand Indicates standard deviation;

[0038] A maximum value aggregation strategy is used to aggregate the response values ​​of all hand keypoints detected in the video frames in a two-dimensional coordinate grid to obtain the guidance feature map. The following relationship exists in the process:

[0039] ;

[0040] in, This indicates a guiding heatmap. Indicates the guiding heatmap in The value at that location, This indicates taking the maximum value. This indicates the total number of key points on the hand.

[0041] Furthermore, the step of fusing the guiding heatmap with the multi-scale fusion feature map using the gated fusion attention module to obtain the spatially refined feature sequence specifically includes the following sub-steps:

[0042] Max pooling is performed on the guiding heatmap to align its spatial resolution with the multi-scale fused feature map, resulting in an aligned heatmap.

[0043] After incrementing the aligned heatmap by 1, it is multiplied element-wise with the multi-scale fused feature map to obtain the attention-modulated feature map. The following relationship exists in the corresponding process:

[0044] ;

[0045] in, This represents the feature map after attention modulation. This represents a multi-scale fused feature map. This represents element-wise multiplication. This represents the aligned heatmap.

[0046] It should be noted that if multiplication fusion is used directly, the original video features will be forcibly set to zero in the background region (i.e., non-hand region) where the guiding heatmap response value is 0. This will lead to the permanent loss of background context information and destroy the integrity of the features. By introducing the "+1" operation, the fusion operation is transformed into the form of "original features + enhanced features", ensuring that the network can still retain the original input features in the worst case and avoid information bottleneck.

[0047] The attention-modulated feature map is concatenated with the multi-scale fused feature map, and then gated fusion is performed through a 1×1×1 convolutional layer to obtain a gated fused feature map. The following relationship exists in the corresponding process:

[0048] ;

[0049] in, Represents the gating fusion feature map. This indicates that the feature has been spliced.

[0050] By performing residual concatenation between the gated fused feature map and the multi-scale fused feature map, a spatially refined feature sequence is obtained. The following relationship exists in the corresponding process:

[0051] ;

[0052] in, This represents a spatially refined feature sequence.

[0053] Step 3: Use the time modeling module to perform feature decomposition, multi-scale time dependency capture, and decoupled feature interaction processing on the spatially refined feature sequence in sequence to obtain the time-varying feature sequence.

[0054] Please see Figure 4 and Figure 5 In step 3, the time modeling module is used to sequentially perform feature decomposition, multi-scale time dependency capture, and decoupled feature interaction processing on the spatially refined feature sequence to obtain the time-varying feature sequence. Specifically, this includes the following sub-steps:

[0055] The spatially refined feature sequence is divided along the channel dimension using variable decomposition units to obtain a set of independent sub-feature vectors;

[0056] The set of independent sub-feature vectors contains M independent sub-feature vectors;

[0057] Based on the set of independent sub-feature vectors, modern convolutional modules are used to perform time modeling on the independent sub-feature vectors to capture multi-scale time dependencies and obtain time-modeled sub-feature vectors.

[0058] By using a decoupled feature interaction network, intra-variable and inter-variable feature interactions are performed on the time-modeled sub-feature vectors to obtain the interacted sub-feature vectors.

[0059] The time-varying feature sequence is obtained by fusing the interacting sub-feature vectors using variable aggregation units.

[0060] It should be noted that, in Figure 5 middle, This represents the number of feature channels; the decoupled feature interaction network includes two orthogonal mixing operations, specifically a two-stage mixing operation: the first stage is intra-variable mixing, which uses grouped pointwise convolution (1×1 kernel size, number of groups = number of variables M) to process the M independent sub-feature vectors obtained from the decomposition in parallel (e.g., Figure 5 As shown in (a)), the information flow is restricted to the channel within each subspace, enabling in-depth mining of a single sub-dynamic (such as wrist posture, finger shape) and avoiding semantic interference between different subspaces; the second stage is variable mixing. After feature rearrangement, another group of pointwise convolutions is used (convolution kernel size 1×1, number of groups = feature dimension D of a single subspace / number of variables M) (as shown in (a)). Figure 5 As shown in (b) of the diagram, cross-subspace information exchange is achieved at each feature dimension, capturing the dependencies and collaborative patterns between different sub-dynamics. This two-stage hybrid design, while maintaining computational efficiency, achieves complex global dependency modeling capabilities comparable to attention mechanisms, effectively solving the problem of insufficient cross-dimensional correlation capture in traditional convolutional networks.

[0061] Furthermore, modern convolutional modules employ a dual-branch depthwise separable convolutional structure, specifically including:

[0062] The large kernel convolution branch is configured with a large kernel to capture long-range dependencies between distant frames in a time series.

[0063] The small kernel convolution branch, configured with a small-sized convolution kernel, is used to extract local high-frequency details between adjacent frames in a time series.

[0064] It should be noted that the dual-branch structure of modern convolutional modules runs independently during model training. By enriching the gradient flow during backpropagation, it effectively addresses the instability issue of large-kernel convolution training and enhances the model's ability to learn multi-scale temporal patterns.

[0065] Step 4: Process the time-varying feature sequence sequentially through average pooling, layer normalization, and fully connected layers to obtain the prediction score matrix.

[0066] In step 4, the time-varying feature sequence is processed sequentially through average pooling, layer normalization, and a fully connected layer to obtain the prediction score matrix. This process includes the following sub-steps:

[0067] The time-varying feature sequence is subjected to average pooling to obtain average pooled features. The following relationship exists in the corresponding process:

[0068] ;

[0069] in, This represents the average pooling characteristic. Indicates the time step. Indicates the index of the time step. Represents time-varying feature sequences;

[0070] The average pooling features are then subjected to layer normalization to obtain the normalized features. The following relationship exists in the corresponding process:

[0071] ;

[0072] in, The features after normalization of the representation layer, This represents the mean of the average pooling feature along the channel dimension. This represents the standard deviation of the average pooling feature along the channel dimension. This represents a small constant used to prevent division by zero. This represents the learnable scaling parameter. Represents the learnable translation parameters;

[0073] By using a fully connected layer to map the normalized features to the class space, a prediction score matrix is ​​obtained. The following relationship exists in the mapping process:

[0074] ;

[0075] in, Represents the predicted score matrix. Represents the weight matrix. Indicates transpose. This represents the bias vector.

[0076] Step 5: Construct the cross-entropy loss function based on the prediction score matrix, and use the cross-entropy loss function to train the lightweight multivariate spatiotemporal convolutional model to obtain the trained lightweight multivariate spatiotemporal convolutional model; input the dynamic gesture video clip into the trained lightweight multivariate spatiotemporal convolutional model to obtain the dynamic gesture recognition result.

[0077] In step 4, the cross-entropy loss function is constructed based on the predicted score matrix, and the following relationship exists in the corresponding process:

[0078] ;

[0079] in, Indicates the first The sample belongs to the first Predicted probability of gestures, Indicates the index of the sample. Index representing the gesture category, This indicates that the data has been processed using the Softmax function. Represents the first in the predicted score matrix The sample belongs to the first Confidence score of gesture-like objects. Represents cross-entropy loss, Indicates the number of samples. Indicates an indicator function, Indicates the first The true label of each sample This indicates taking the logarithm.

[0080] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0081] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0082] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0083] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution, characterized in that, The method includes the following steps: Step 1: Construct a spatial feature extraction module based on a pseudo-3D gated attention fusion network, and a temporal modeling module based on a multivariate decomposition temporal convolutional network. The spatial feature extraction module and the temporal modeling module together form a lightweight multivariate spatiotemporal convolutional model. The pseudo-3D gated attention fusion network includes: a basic feature extraction layer, an Inception Block module, and a gated fusion attention module. The multivariate decomposition temporal convolutional network includes: a variable decomposition unit, a modern convolutional module, a decoupled feature interaction network, and a variable aggregation unit. Step 2: Acquire dynamic gesture video clips, and use the spatial feature extraction module to sequentially perform multi-scale spatial feature extraction and prior enhancement processing on the dynamic gesture video clips to obtain a spatially refined feature sequence. This includes the following sub-steps: A preliminary feature map is obtained by using a basic feature extraction layer to perform preliminary feature encoding on dynamic gesture video clips; The Inception Block module is used to perform multi-scale feature fusion on the preliminary feature map to obtain a multi-scale fused feature map. The Inception Block module includes four parallel convolutional branches. Based on dynamic gesture video clips, a guidance heatmap is obtained. The guidance heatmap is then fused with a multi-scale fusion feature map using a gated fusion attention module to obtain a spatially refined feature sequence. Step 3: Using the time modeling module, perform feature decomposition, multi-scale time dependency capture, and decoupled feature interaction processing on the spatially refined feature sequence to obtain the time-varying feature sequence. This includes the following sub-steps: The spatially refined feature sequence is divided along the channel dimension using variable decomposition units to obtain a set of independent sub-feature vectors; Based on the set of independent sub-feature vectors, modern convolutional modules are used to perform time modeling on the independent sub-feature vectors to capture multi-scale time dependencies and obtain time-modeled sub-feature vectors. By using a decoupled feature interaction network, intra-variable and inter-variable feature interactions are performed on the time-modeled sub-feature vectors to obtain the interacted sub-feature vectors. The time-varying feature sequence is obtained by fusing the interacting sub-feature vectors using variable aggregation units. The modern convolutional module adopts a dual-branch depthwise separable convolutional structure, wherein the dual branches include a large kernel convolutional branch and a small kernel convolutional branch. Step 4: Process the time-varying feature sequence sequentially through average pooling, layer normalization, and fully connected layers to obtain the prediction score matrix; Step 5: Construct the cross-entropy loss function based on the prediction score matrix, and use the cross-entropy loss function to train the lightweight multivariate spatiotemporal convolutional model to obtain the trained lightweight multivariate spatiotemporal convolutional model; input the dynamic gesture video clip into the trained lightweight multivariate spatiotemporal convolutional model to obtain the dynamic gesture recognition result.

2. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 1, characterized in that, The step of using the Inception Block module to perform multi-scale feature fusion on the preliminary feature map to obtain a multi-scale fused feature map specifically includes the following sub-steps: The initial feature maps are assigned to four parallel convolutional branches in the Inception Block module for processing to obtain the first branch features, the second branch features, the third branch features, and the fourth branch features; The first convolutional branch is a 1×1×1 convolutional layer, the second convolutional branch includes a 1×1×1 convolutional layer and a 1×3×3 convolutional layer in sequence, the third convolutional branch includes a 1×1×1 convolutional layer and a 1×3×3 convolutional layer in sequence, and the fourth convolutional branch includes a max pooling layer and a 1×1×1 convolutional layer in sequence. The features from the first branch, the second branch, the third branch, and the fourth branch are concatenated to obtain a multi-scale fused feature map.

3. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 2, characterized in that, The step of obtaining the guide heatmap based on dynamic gesture video clips specifically includes the following sub-steps: A two-dimensional coordinate grid is constructed based on video frames of dynamic gesture video clips. The response values ​​of hand key points detected in the video frames in the two-dimensional coordinate grid are calculated using a two-dimensional Gaussian kernel function. The following relationship exists in the correspondence process: ； in, Indicates the first [number]th ... The first detected in the frame Key hand features are represented in a two-dimensional coordinate grid. The response value at that location, Indicates the index of the video frame. Indexes representing key points of the hand. Represents pixel coordinates in a two-dimensional coordinate grid. Represents an exponential function. Indicates the first [number]th ... The first detected in the frame Coordinates of key points on the hand Indicates standard deviation; A maximum value aggregation strategy is used to aggregate the response values ​​of all hand keypoints detected in the video frames in a two-dimensional coordinate grid to obtain a guidance heatmap. The following relationship exists in the process: ； wherein, represents a guided heat map, represents a value of the guided heat map at represents taking a maximum value, represents taking a maximum value, represents a total number of hand key points at 4. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 3, characterized in that, The step of fusing the guiding heatmap and the multi-scale fusion feature map using the gated fusion attention module to obtain the spatially refined feature sequence specifically includes the following sub-steps: Max pooling is performed on the guiding heatmap to align its spatial resolution with the multi-scale fused feature map, resulting in an aligned heatmap. After incrementing the aligned heatmap by 1, it is multiplied element-wise with the multi-scale fused feature map to obtain the attention-modulated feature map. The attention-modulated feature map is concatenated with the multi-scale fused feature map, and then gated fusion is performed through a 1×1×1 convolutional layer to obtain a gated fused feature map. By performing residual concatenation between the gated fusion feature map and the multi-scale fusion feature map, a spatially refined feature sequence is obtained.

5. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 4, characterized in that, In the step of incrementing the aligned heatmap by 1 and then multiplying it element-wise with the multi-scale fused feature map to obtain the attention-modulated feature map, the following relationship exists: ； in, This represents the feature map after attention modulation. This represents a multi-scale fused feature map. This represents element-wise multiplication. This represents the aligned heatmap. In the step of concatenating the attention-modulated feature map with the multi-scale fused feature map and then performing gated fusion through a 1×1×1 convolutional layer to obtain the gated fused feature map, the following relationship exists: ； wherein, denotes a gating fusion feature map, denotes a feature splicing process; In the step of residually connecting the gated fused feature map and the multi-scale fused feature map to obtain the spatially refined feature sequence, the following relationship exists: ； wherein represents a sequence of spatial refinement features.

6. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 5, characterized in that, The modern convolutional module employs a dual-branch depthwise separable convolutional structure, specifically including: The large kernel convolution branch is configured with a large kernel to capture long-range dependencies between distant frames in a time series. The small kernel convolution branch, configured with a small-sized convolution kernel, is used to extract local high-frequency details between adjacent frames in a time series.

7. The dynamic gesture recognition method based on lightweight multivariate spatio-temporal convolution according to claim 6, characterized in that, In step 4, the time-varying feature sequence is processed sequentially through average pooling, layer normalization, and a fully connected layer to obtain the prediction score matrix. This process includes the following sub-steps: The time-varying feature sequence is subjected to average pooling to obtain average pooled features. The following relationship exists in the corresponding process: ； wherein, denotes the average-pooled features, denotes the time step, denotes the index of the time step, denotes the time-varying feature sequence; The average pooling features are then subjected to layer normalization to obtain the normalized features. The following relationship exists in the corresponding process: ； in, The features after normalization of the representation layer, This represents the mean of the average pooling feature along the channel dimension. This represents the standard deviation of the average pooling feature along the channel dimension. This represents a small constant used to prevent division by zero. This represents the learnable scaling parameter. Represents the learnable translation parameters; By using a fully connected layer to map the normalized features to the class space, a prediction score matrix is ​​obtained. The following relationship exists in the mapping process: ； in, Represents the predicted score matrix. Represents the weight matrix. Indicates transpose. This represents the bias vector.

8. The dynamic gesture recognition method based on lightweight multivariate spatiotemporal convolution according to claim 7, characterized in that, In step 5, the cross-entropy loss function is constructed based on the predicted score matrix, and the following relationship exists in the corresponding process: ； in, Indicates the first The sample belongs to the first Predicted probability of gestures, Indicates the index of the sample. Index representing the gesture category, This indicates that the data has been processed using the Softmax function. Represents the first in the predicted score matrix The sample belongs to the first Confidence score of gesture-like objects. Represents cross-entropy loss, Indicates the number of samples. Indicates an indicator function, Indicates the first The true label of each sample This indicates taking the logarithm.

Images