Dynamic gesture detection method and system and electronic equipment
By combining the spatiotemporal Transformer encoder and the ST-GCN architecture, the shortcomings of temporal modeling and feature fusion in dynamic gesture detection are addressed, achieving gesture detection with higher accuracy and robustness.
Patent Information
- Application Number
- CN202511762731.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-24
AI Technical Summary
Existing dynamic gesture detection methods lack the ability to perform temporal modeling, making it difficult to capture the dependencies between long-distance frames in long sequences. Furthermore, their feature fusion strategies are simple and cannot adapt to complex scenarios, resulting in poor performance.
We adopt an architecture that combines a spatiotemporal Transformer encoder and a spatiotemporal graph convolutional network (ST-GCN). We model global spatiotemporal dependencies through a self-attention mechanism and design a cross-modal attention fusion module to adaptively adjust the feature contribution weights.
It significantly improves the accuracy and robustness of dynamic gesture detection in complex scenarios, enabling more precise characterization of subtle gesture movements and enhancing the model's generalization and anti-interference capabilities.
Smart Images
Figure CN121564798A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a dynamic gesture detection method and system, and electronic equipment. Background Technology
[0002] Dynamic gesture detection is a core research direction in the fields of human-computer interaction and computer vision. Its accuracy directly affects the interactive experience and reliability of applications such as virtual reality control, intelligent vehicle systems, and industrial automation control. With the development of deep learning technology, dynamic gesture recognition methods have undergone a significant evolution from traditional manual feature-driven models to modern data-driven models. Early research mainly relied on traditional algorithms such as Histogram of Oriented Gradients (HOG) and Dynamic Time Warping (DTW). Although these algorithms have the advantage of high computational efficiency, their feature representation capabilities are limited, making it difficult to adapt to complex backgrounds and diverse gesture styles. In recent years, end-to-end methods represented by deep learning have brought new breakthroughs to this field, with dynamic gesture recognition methods based on two-stream convolutional architectures being the most representative.
[0003] This type of dual-stream architecture method fuses appearance video frames and hand keypoint sequences, employs a 3D convolutional network (3D CNN) to extract spatiotemporal features from the video, and utilizes a recurrent neural network (RNN) to model the motion trajectory of the keypoints. Finally, classification is achieved through feature concatenation or weighted averaging. This approach represents the mainstream technical route for current dynamic gesture recognition. Its advantage lies in simultaneously considering both appearance texture information and motion structure information, exhibiting superior performance compared to traditional methods under controlled environments, and providing an important foundation for gesture recognition research.
[0004] However, this type of technical solution still suffers from a series of fundamental flaws. First, its temporal modeling capability is limited; RNN-like models struggle to effectively capture long-distance inter-frame dependencies in long sequences and suffer from the vanishing gradient problem, resulting in insufficient modeling ability for long-term gestures. Second, the model's utilization of keypoint structural information is crude, simply serializing keypoint coordinates as input, thus disrupting the inherent topological structure of the hand and failing to explicitly model the complex motion constraints between joints. More importantly, the feature fusion strategies of existing methods are too simplistic, unable to adaptively adjust the contribution weights of multimodal features based on the characteristics of the input samples, and lack robust mechanisms for handling noisy keypoint data. These flaws severely limit its performance in complex real-world scenarios, necessitating a novel technical solution to address these fundamental problems. Summary of the Invention
[0005] This invention provides a dynamic gesture detection method, system, and electronic device to address the shortcomings of related technologies in terms of poor temporal modeling capabilities and poor performance in complex scenarios when performing dynamic gesture detection. The solution of this application provides a gesture detection method with higher accuracy.
[0006] This invention provides a dynamic gesture detection method, comprising: Obtain the target video sequence information to be detected; The target video sequence information is preprocessed to obtain an appearance frame sequence and a key point sequence; The appearance frame sequence is processed using a spatiotemporal Transformer encoder to obtain an appearance feature vector, and the key point sequence is processed using a spatiotemporal Transformer encoder to obtain a key point feature vector. Cross-modal attention fusion is performed on the appearance feature vector and the key point feature vector to generate a fused feature vector; Dynamic gesture detection is performed based on the fused feature vector.
[0007] According to the dynamic gesture detection method provided by the present invention, the preprocessing of the target video sequence information to obtain an appearance frame sequence and a key point sequence includes: After center cropping and scaling the target video sequence, pixel values are normalized to obtain the appearance frame sequence. Key points are extracted from the target video sequence, and the extracted key points are normalized to obtain a key point sequence.
[0008] According to the dynamic gesture detection method provided by the present invention, the step of processing the appearance frame sequence based on a spatiotemporal Transformer encoder to obtain an appearance feature vector includes: Visual features are obtained by extracting features from the appearance frame sequence using a convolutional neural backbone network. The visual features are input into a spatiotemporal Transformer encoder, and a global spatiotemporal dependency is established based on a self-attention mechanism to obtain an appearance feature vector.
[0009] According to the dynamic gesture detection method provided by the present invention, the step of processing the key point sequence based on the spatiotemporal Transformer encoder to obtain key point feature vectors includes: Motion features are obtained by extracting features from the keypoint sequence using a spatiotemporal graph convolutional network. The motion features are input into the spatiotemporal Transformer encoder, and the appearance feature vector is obtained based on the self-attention mechanism.
[0010] According to the dynamic gesture detection method provided by the present invention, the step of performing cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector includes: Calculate the cross-attention weights between the appearance feature vector and the key point feature vector; Based on the cross-attention weights, the appearance feature vector and the key point feature vector are adaptively weighted and fused. The weighted fused features are residually connected with the key point features to obtain the fused feature vector.
[0011] According to the dynamic gesture detection method provided by the present invention, the step of performing dynamic gesture detection based on the fused feature vector includes: The probability distribution of each gesture type is calculated using a fully connected classifier; Based on the probability distribution, the gesture corresponding to the target video sequence information is determined.
[0012] According to the dynamic gesture detection method provided by the present invention, the step of calculating the probability distribution of each gesture type using a fully connected classifier includes: The fused feature vector is input into a fully connected classifier for nonlinear transformation; The probability distribution for each gesture category is calculated using the Softmax activation function.
[0013] According to the dynamic gesture detection method provided by the present invention, determining the gesture corresponding to the target video sequence information based on the probability distribution includes: From the probability distribution, the gesture with the highest probability is selected as the gesture corresponding to the target video sequence information.
[0014] The present invention also provides a dynamic gesture detection system, comprising: The information acquisition module is used to acquire the target video sequence information to be detected; The preprocessing module is used to preprocess the target video sequence information to obtain an appearance frame sequence and a key point sequence; The vector transformation module is used to process the appearance frame sequence based on the spatiotemporal Transformer encoder to obtain appearance feature vectors, and to process the key point sequence based on the spatiotemporal Transformer encoder to obtain key point feature vectors. The vector fusion module is used to perform cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector. The gesture detection module is used to perform dynamic gesture detection based on the fused feature vector.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the dynamic gesture detection methods described above.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the dynamic gesture detection methods described above.
[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the dynamic gesture detection methods described above.
[0018] The dynamic gesture detection method provided by this invention employs a Transformer encoder in both the appearance and keypoint branches, utilizing its global self-attention mechanism to replace the traditional RNN / 3D CNN, thus completely overcoming the bottleneck of long-range temporal dependency modeling. It innovatively introduces a spatiotemporal graph convolutional network (ST-GCN) into the keypoint stream, explicitly extracting the topological structure and motion constraints between hand joints before hand-to-hand global temporal modeling, achieving refined perception of structured gestures. An attention-based adaptive fusion module is designed, which dynamically calibrates the contribution weights of the dual-stream features based on the input content, thereby significantly improving the robustness and accuracy of the model in complex scenarios. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is one of the flowcharts of the dynamic gesture detection method provided in the embodiments of the present invention; Figure 2 This is a second schematic flowchart of the dynamic gesture detection method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the data preprocessing process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the feature vector extraction process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the dynamic gesture detection system provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0022] Figure 1 This is one of the flowcharts of the dynamic gesture detection method provided in the embodiments of the present invention.
[0023] Figure 2 This is the second flowchart of the dynamic gesture detection method provided in this embodiment of the invention.
[0024] like Figure 1 and Figure 2 As shown, this embodiment provides a dynamic gesture detection method, including: Step 101: Obtain the target video sequence information to be detected; Step 102: Preprocess the target video sequence information to obtain an appearance frame sequence and a key point sequence; In practical applications, the target video sequence can be center-cropped and scaled before pixel value normalization to obtain the appearance frame sequence. Key points are extracted from the target video sequence. After normalizing the extracted key points, a key point sequence is obtained. The key point sequence may include the coordinates of 21 hand joints.
[0025] Figure 3 This is a schematic diagram of the data preprocessing process provided in an embodiment of the present invention.
[0026] like Figure 3 As shown, specifically, the original target video sequence information can be... ; Each frame of the target video sequence can be centrally cropped and scaled to a fixed size. Then, pixel values are normalized to obtain the appearance frame sequence; Each frame of the image can be processed using a pre-trained hand keypoint detection model to extract the two-dimensional coordinate data of 21 hand joints. ,in Furthermore, the key point coordinate data is normalized to eliminate the influence of hand size variations and image position, thus obtaining a key point sequence. After obtaining the appearance frame sequence and keypoint sequence, the appearance frame sequence and keypoint sequence can be aligned according to the timestamp to ensure that data at the same time point corresponds to the same frame image, and the preprocessed standardized bimodal dataset can be output. .
[0027] After preprocessing, a time-aligned appearance frame sequence and key point sequence can be formed as a dual-modal input.
[0028] Step 103: The appearance frame sequence is processed by the spatiotemporal Transformer encoder to obtain the appearance feature vector, and the key point sequence is processed by the spatiotemporal Transformer encoder to obtain the key point feature vector. In practical applications, visual features can be obtained by extracting features from the appearance frame sequence using a convolutional neural backbone network. The visual features are input into the spatiotemporal Transformer encoder, and a global spatiotemporal dependency is established based on the self-attention mechanism to obtain the appearance feature vector; Motion features are obtained by extracting features from the keypoint sequence using a spatiotemporal graph convolutional network. The motion features are input into the spatiotemporal Transformer encoder, and the appearance feature vector is obtained based on the self-attention mechanism.
[0029] In practical applications, the preprocessed bimodal data can be input into two parallel deep feature extraction branches. The appearance stream branch uses a 2D CNN backbone network to extract frame-level spatial features, then models global spatiotemporal dependencies using a spatiotemporal Transformer encoder, outputting an appearance feature vector. The keypoint stream branch first uses a temporal graph convolutional network (ST-GCN) to extract local topological structures and motion features between hand joints, then captures the global temporal context using a temporal Transformer encoder, outputting a keypoint feature vector.
[0030] Figure 4 This is a schematic diagram of the feature vector extraction process provided in an embodiment of the present invention.
[0031] Specifically, when extracting appearance feature vectors and key point feature vectors, one can, as follows: Figure 4 As shown, the sequence of appearance frames is input into a 2D CNN backbone network, and hierarchical visual features of each frame are extracted through depthwise convolution operations. ; The CNN features are input into the spatiotemporal Transformer encoder, which models the global spatiotemporal dependencies between and within frames through a self-attention mechanism, and outputs an appearance feature vector. ; The keypoint sequence is input into the spatiotemporal graph convolutional network (ST-GCN), and the local topological structure and motion features between hand joints are extracted through graph convolution operations. ; The ST-GCN features are input into the temporal Transformer encoder, which captures global temporal context information through a self-attention mechanism and outputs keypoint feature vectors. ; Output the bimodal high-level feature set of all samples. .
[0032] Step 104: Perform cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector; In practical applications, the cross-attention weights between the appearance feature vector and the key point feature vector can be calculated; Based on the cross-attention weights, the appearance feature vector and the key point feature vector are adaptively weighted and fused. The weighted fused features are residually connected with the key point features to obtain the fused feature vector.
[0033] Specifically, a set of bimodal features can be selected. They are projected into the query space, key space, and value space, respectively. Calculate the cross-attention weights between appearance features and keypoint features: ,in These are the learnable query space matrix and the key space matrix, respectively. Adaptive weighted fusion of features based on attention weights: ,in It is a learnable value space matrix; The weighted features are residually concatenated with the original keypoint features to obtain deep fusion features. ; Repeat the above steps until all bimodal features have been processed, and output the fused feature set. .
[0034] Step 105: Perform dynamic gesture detection based on the fused feature vector.
[0035] In practice, a fully connected classifier can be used to calculate the probability distribution of each type of gesture; Based on the probability distribution, the gesture corresponding to the target video sequence information is determined; The process involves using a fully connected classifier to calculate the probability distribution for each gesture type, including: The fused feature vector is input into a fully connected classifier for nonlinear transformation; The probability distribution for each gesture category is calculated using the Softmax activation function.
[0036] Based on the probability distribution, determining the gesture corresponding to the target video sequence information includes: From the probability distribution, the gesture with the highest probability is selected as the gesture corresponding to the target video sequence information.
[0037] Specifically, the fused features can be input into a fully connected network and subjected to a nonlinear transformation: ; The probability distribution for each gesture category is calculated using the Softmax function: ; Select the category with the highest probability as the final gesture recognition result and output it: ; During training, the cross-entropy loss between the predicted result and the true label can also be calculated: The parameters of the entire network are optimized through backpropagation algorithm, and the trained dynamic gesture detection model is output.
[0038] Compared with existing technical solutions, the dynamic gesture detection method provided in this application has the following advantages: 1. Existing technical solutions mostly use 3D CNN or CNN+RNN for temporal modeling, which makes it difficult to effectively capture the global dependencies between long-distance frames in long-sequence gestures. This invention innovatively introduces Transformer encoders in both stream branches. With the help of its global self-attention mechanism, it can directly model the long-range contextual information between any image blocks in the appearance sequence and between any time steps in the keypoint sequence, thereby significantly improving the modeling accuracy and robustness of complex long-sequence gestures.
[0039] 2. Existing methods typically perform simple serialization of hand keypoint coordinates, destroying their inherent topological structure and failing to explicitly model the motion constraints between joints. This invention innovatively employs a cascaded architecture of Spatiotemporal Graph Convolution (ST-GCN) and Temporal Transformer in the keypoint flow branch. First, ST-GCN is used to extract local structural features and motion patterns between hand joints, and then the Transformer captures global temporal dependencies, thereby fully utilizing the prior graph structure of the hand to more accurately characterize the subtle movements of the gesture.
[0040] 3. Existing dual-stream fusion methods mostly rely on simple feature concatenation or weighted averaging, which is a static and shallow fusion strategy that cannot adaptively adjust modal weights according to the dynamic characteristics of the input samples. This invention designs an attention-based feature fusion module that can dynamically calculate the correlation between appearance feature vectors and keypoint feature vectors, and accordingly achieve adaptive weighted fusion. This intelligently selects more reliable feature sources when keypoint detection is noisy or the appearance background is complex, greatly enhancing the model's generalization ability and anti-interference capability.
[0041] The dynamic gesture detection system provided by the present invention is described below. The dynamic gesture detection system described below can be referred to in correspondence with the dynamic gesture detection method described above.
[0042] Figure 5 This is a schematic diagram of the dynamic gesture detection system provided in an embodiment of the present invention. Figure 5 As shown, the dynamic gesture detection system includes: Information acquisition module 501 is used to acquire the target video sequence information to be detected; Preprocessing module 502 is used to preprocess the target video sequence information to obtain an appearance frame sequence and a key point sequence; The vector transformation module 503 is used to process the appearance frame sequence based on the spatiotemporal Transformer encoder to obtain appearance feature vectors, and to process the key point sequence based on the spatiotemporal Transformer encoder to obtain key point feature vectors. The vector fusion module 504 is used to perform cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector. The gesture detection module 505 is used to perform dynamic gesture detection based on the fused feature vector.
[0043] In an exemplary embodiment, the preprocessing module is further configured to: perform center cropping and scaling on the target video sequence and then normalize the pixel values to obtain an appearance frame sequence; Key points are extracted from the target video sequence, and the extracted key points are normalized to obtain a key point sequence.
[0044] In an exemplary embodiment, the vector transformation module is further configured to: extract features from the appearance frame sequence through a convolutional neural backbone network to obtain visual features; The visual features are input into a spatiotemporal Transformer encoder, and a global spatiotemporal dependency is established based on a self-attention mechanism to obtain an appearance feature vector.
[0045] In an exemplary embodiment, the vector transformation module is further configured to: extract features from the keypoint sequence using a spatiotemporal graph convolutional network to obtain motion features; The motion features are input into the spatiotemporal Transformer encoder, and the appearance feature vector is obtained based on the self-attention mechanism.
[0046] In an exemplary embodiment, the vector fusion module is further configured to: calculate the cross-attention weights between the appearance feature vector and the key point feature vector; Based on the cross-attention weights, the appearance feature vector and the key point feature vector are adaptively weighted and fused. The weighted fused features are residually connected with the key point features to obtain the fused feature vector.
[0047] In an exemplary embodiment, the gesture detection module is further configured to: calculate the probability distribution of each type of gesture using a fully connected classifier; Based on the probability distribution, determine the hand corresponding to the target video sequence information. In an exemplary embodiment, the gesture detection module is further configured to: input the fused feature vector into a fully connected classifier for nonlinear transformation; The probability distribution for each gesture category is calculated using the Softmax activation function.
[0048] In an exemplary embodiment, the gesture detection module is further configured to: select the gesture with the highest probability from the probability distribution as the gesture corresponding to the target video sequence information.
[0049] The specific implementation method of the dynamic gesture detection system provided in this embodiment can be implemented with reference to the above embodiment, and will not be repeated here.
[0050] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a dynamic gesture detection method, which includes: Obtain the target video sequence information to be detected; The target video sequence information is preprocessed to obtain an appearance frame sequence and a key point sequence; The appearance frame sequence is processed using a spatiotemporal Transformer encoder to obtain an appearance feature vector, and the key point sequence is processed using a spatiotemporal Transformer encoder to obtain a key point feature vector. Cross-modal attention fusion is performed on the appearance feature vector and the key point feature vector to generate a fused feature vector; Dynamic gesture detection is performed based on the fused feature vector.
[0051] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the dynamic gesture detection method provided by the above methods, the method including: Obtain the target video sequence information to be detected; The target video sequence information is preprocessed to obtain an appearance frame sequence and a key point sequence; The appearance frame sequence is processed using a spatiotemporal Transformer encoder to obtain an appearance feature vector, and the key point sequence is processed using a spatiotemporal Transformer encoder to obtain a key point feature vector. Cross-modal attention fusion is performed on the appearance feature vector and the key point feature vector to generate a fused feature vector; Dynamic gesture detection is performed based on the fused feature vector.
[0053] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic gesture detection method provided by the methods described above, the method comprising: Obtain the target video sequence information to be detected; The target video sequence information is preprocessed to obtain an appearance frame sequence and a key point sequence; The appearance frame sequence is processed using a spatiotemporal Transformer encoder to obtain an appearance feature vector, and the key point sequence is processed using a spatiotemporal Transformer encoder to obtain a key point feature vector. Cross-modal attention fusion is performed on the appearance feature vector and the key point feature vector to generate a fused feature vector; Dynamic gesture detection is performed based on the fused feature vector.
[0054] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A dynamic gesture detection method, characterized in that, include: Obtain the target video sequence information to be detected; The target video sequence information is preprocessed to obtain an appearance frame sequence and a key point sequence; The appearance frame sequence is processed using a spatiotemporal Transformer encoder to obtain an appearance feature vector, and the key point sequence is processed using a spatiotemporal Transformer encoder to obtain a key point feature vector. Cross-modal attention fusion is performed on the appearance feature vector and the key point feature vector to generate a fused feature vector; Dynamic gesture detection is performed based on the fused feature vector.
2. The dynamic gesture detection method according to claim 1, characterized in that, The preprocessing of the target video sequence information to obtain the appearance frame sequence and key point sequence includes: After center cropping and scaling the target video sequence, pixel values are normalized to obtain the appearance frame sequence. Key points are extracted from the target video sequence, and the extracted key points are normalized to obtain a key point sequence.
3. The dynamic gesture detection method according to claim 1, characterized in that, The process of obtaining appearance feature vectors by processing the appearance frame sequence using a spatiotemporal Transformer encoder includes: Visual features are obtained by extracting features from the appearance frame sequence using a convolutional neural backbone network. The visual features are input into a spatiotemporal Transformer encoder, and a global spatiotemporal dependency is established based on a self-attention mechanism to obtain an appearance feature vector.
4. The dynamic gesture detection method according to claim 1, characterized in that, The process of obtaining keypoint feature vectors by processing the keypoint sequence using a spatiotemporal Transformer encoder includes: Motion features are obtained by extracting features from the keypoint sequence using a spatiotemporal graph convolutional network. The motion features are input into the spatiotemporal Transformer encoder, and the appearance feature vector is obtained based on the self-attention mechanism.
5. The dynamic gesture detection method according to claim 1, characterized in that, The step of performing cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector includes: Calculate the cross-attention weights between the appearance feature vector and the key point feature vector; Based on the cross-attention weights, the appearance feature vector and the key point feature vector are adaptively weighted and fused. The weighted fused features are residually connected with the key point features to obtain the fused feature vector.
6. The dynamic gesture detection method according to claim 1, characterized in that, The dynamic gesture detection based on the fused feature vector includes: The probability distribution of each gesture type is calculated using a fully connected classifier; Based on the probability distribution, the gesture corresponding to the target video sequence information is determined.
7. The dynamic gesture detection method according to claim 6, characterized in that, The step of calculating the probability distribution for each gesture type using a fully connected classifier includes: The fused feature vector is input into a fully connected classifier for nonlinear transformation; The probability distribution for each gesture category is calculated using the Softmax activation function.
8. The dynamic gesture detection method according to claim 6, characterized in that, The step of determining the gesture corresponding to the target video sequence information based on the probability distribution includes: From the probability distribution, the gesture with the highest probability is selected as the gesture corresponding to the target video sequence information.
9. A dynamic gesture detection system, characterized in that, include: The information acquisition module is used to acquire the target video sequence information to be detected; The preprocessing module is used to preprocess the target video sequence information to obtain an appearance frame sequence and a key point sequence; The vector transformation module is used to process the appearance frame sequence based on the spatiotemporal Transformer encoder to obtain appearance feature vectors, and to process the key point sequence based on the spatiotemporal Transformer encoder to obtain key point feature vectors. The vector fusion module is used to perform cross-modal attention fusion on the appearance feature vector and the key point feature vector to generate a fused feature vector. The gesture detection module is used to perform dynamic gesture detection based on the fused feature vector.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the dynamic gesture detection method as described in any one of claims 1-8.