Iterative diffusion gesture recognition method and system based on memory guidance and prediction verification

Abstract

The invention relates to the field of human-computer interaction, virtual reality and augmented reality, and discloses an iterative diffusion gesture recognition method and system based on memory guidance and prediction verification, and the method comprises the steps: constructing a multi-scale feature extraction network; performing initial gesture state prediction and feature completion operation on the multi-scale spatial-temporal features based on a historical memory library to obtain generated initial completion features and future frame gesture features; the diffusion probability model is guided to execute multi-step reverse denoising on the hand shielding area, and repair features of the hand shielding area are synchronously generated; calculating a comprehensive confidence coefficient based on the repair features and a biomechanical prediction verification result; and performing action classification based on the final repair features of the hand occlusion area, and outputting a gesture action classification result. According to the method, a minimum generation forest topology admission mechanism based on key point uncertainty propagation is constructed, and by calculating the Bayesian probability weighted distance, a limb fracture area caused by shielding can be automatically identified and isolated.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, virtual reality, and augmented reality, and more specifically, to an iterative diffusion gesture recognition method and system based on memory guidance and prediction verification. Background Technology

[0002] Gesture recognition is a technology that uses computer vision, sensors, or other techniques to detect, track, and interpret human gestures, converting them into machine-understandable commands. As a core technology in human-computer interaction, virtual reality, and augmented reality, it faces a severe problem of hand occlusion in practical applications. Existing technologies learn spatiotemporal features directly from video sequences using 3D convolutional neural networks. These methods heavily rely on complete and clear visual information about the hand. When the hand is severely obscured by objects, clothing, or other body parts, the lack of input information leads to a sharp decline in model performance. They inherently lack the ability to intelligently reason about and complete features in invisible areas, failing to recover complete and reasonable gesture features from partially visible information.

[0003] Meanwhile, existing technologies disclose dynamic gesture recognition methods based on hand keypoint detection. These methods first detect 2D / 3D keypoints of the hand and then classify gestures based on the keypoint sequence. Under slight occlusion, these methods can maintain some keypoint detection through spatiotemporal context or model robustness. However, when severe occlusion causes continuous failures in keypoint detection, the recognition chain breaks down, and accuracy drops significantly. Existing restoration methods (such as linear interpolation and mean filling) are too simplistic and cannot generate reasonable pose sequences that conform to hand kinematics and anatomical constraints, resulting in unnatural and discontinuous movements after restoration.

[0004] Furthermore, existing technologies have disclosed image inpainting methods based on Generative Adversarial Networks (GANs). These methods attempt to use GANs to generate and fill in occluded image regions. While GANs have made progress in image inpainting, they suffer from inherent problems such as training instability and mode collapse. When applied to gesture video sequences, it is difficult to guarantee the continuity and consistency of the inpainting results in the spatiotemporal dimensions, which may lead to flickering or jitter in the inpainted frames. More importantly, GANs typically perform a one-time forward generation, lacking an intelligent iterative refinement mechanism to gradually improve the inpainting quality.

[0005] In summary, existing technical solutions are mostly isolated and unidirectional processing flows, lacking a collaborative reasoning framework that combines historical memory, future prediction, and iterative optimization. Specifically, this manifests in the following ways:

[0006] Lack of memory mechanism and topological constraints: It fails to effectively store long-term historical gesture patterns and lacks a mechanism to verify the topological stability of memory samples, which easily introduces noisy samples to pollute the memory bank.

[0007] Lack of prediction verification: Failure to generate reasonable future poses through motion prediction and use them as a basis for verifying the current inference results.

[0008] Lack of adaptive iteration: It cannot dynamically adjust the inference depth based on the self-confidence of the current recognition result, so as to achieve an adaptive balance between computing resources and recognition accuracy.

[0009] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0010] To address the problems in related technologies, this invention proposes an iterative diffusion gesture recognition method and system based on memory guidance and prediction verification, in order to overcome the aforementioned technical problems existing in the prior art.

[0011] Therefore, the specific technical solution adopted by the present invention is as follows:

[0012] According to one aspect of the present invention, an iterative diffusion gesture recognition method based on memory guidance and prediction verification is provided, the method comprising:

[0013] S1. An improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on the convolution kernel and stride parameters of the preset specifications. The hand occlusion video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information are output.

[0014] S2. Based on the historical memory bank, perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features to obtain preliminary completed features and future frame gesture features. Construct a hand biomechanical energy model that includes tendon tension and kinetic energy consumption to physically verify the future frame gesture features and obtain biomechanical prediction verification results.

[0015] S3. Multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features and scene depth perception features are fused into a conditional vector to guide the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area and simultaneously generate repair features for the hand occlusion area.

[0016] S4. Calculate the overall confidence level based on the repair features and biomechanical prediction verification results; if the overall confidence level does not reach the preset threshold, update the historical memory bank based on the intermediate results generated in the current iteration and return to step S2 to carry out a new round of prediction and diffusion process until the iteration conditions are met and stop and proceed to step S5; if the overall confidence level reaches the preset threshold, proceed directly to step S5.

[0017] S5. After the iteration is completed, the action classification is performed based on the final repair features of the hand occlusion area, and the gesture action classification result is output.

[0018] Preferably, the improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on the pre-defined convolutional kernel and stride parameters. The hand occlusion video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information include:

[0019] Extract the RGB video sequence from the hand-occluded video and perform preprocessing;

[0020] The preprocessed RGB video sequence is input into the improved deep residual network. The first layer of the improved deep residual network performs a preliminary spatiotemporal convolution operation based on the preset three-dimensional convolution kernel and stride parameters to extract the initial spatiotemporal feature map.

[0021] The initial spatiotemporal feature map is input into the subsequent layers of the improved deep residual network. The subsequent layers of the improved deep residual network perform two-dimensional spatial convolution and one-dimensional temporal convolution operations in sequence based on spatial convolution kernels and temporal convolution kernels of preset specifications, and output spatiotemporal feature maps of different scales.

[0022] By fusing features from spatiotemporal feature maps of different scales along the time dimension, multi-scale spatiotemporal features containing hand occlusion information are obtained.

[0023] Preferably, the feature fusion of spatiotemporal feature maps of different scales in the time dimension to obtain multi-scale spatiotemporal features containing hand occlusion information includes:

[0024] Spatiotemporal feature maps of different scales are flattened along the time dimension into feature sequences, and corresponding query vectors, key vectors, and value vectors are generated for each element in the feature sequence.

[0025] The attention weight distribution is calculated based on the query vector and key vector, and the value vectors of all elements are weighted and summed according to the attention weight distribution to output temporal fusion feature maps at different scales.

[0026] Upsampling is performed on each temporal fusion feature map to adjust the spatial size to a uniform size; the temporal fusion feature maps with uniform spatial size are then stitched together along the channel dimension to generate multi-scale spatiotemporal features that include hand occlusion information.

[0027] Preferably, the initial gesture state prediction and feature completion operation based on the historical memory bank for multi-scale spatiotemporal features is performed to obtain preliminary completed features and future frame gesture features. A hand biomechanical energy model including tendon tension and kinetic energy consumption is constructed to physically verify the future frame gesture features, and the biomechanical prediction verification results include:

[0028] High-level semantic compression is performed on multi-scale spatiotemporal features, and approximate retrieval of the compressed feature vectors is performed in the semantic layer of the historical memory bank to obtain several candidate memories.

[0029] In the topological layer of the historical memory bank, topological matching is performed based on the minimum spanning forest structure of candidate memories to select memory samples; the complete spatiotemporal features of the selected memory samples are weighted and fused with uncertainty perception to generate preliminary completion features.

[0030] A bidirectional interactive Transformer decoder is constructed. The preliminary completed features are input into the bidirectional interactive Bayesian Transformer decoder to predict the probability distribution of gesture states in future frames and generate gesture features for future frames.

[0031] A biomechanical energy model of the hand, incorporating tendon tension and kinetic energy consumption, is constructed. The hand energy cost function value of gesture features in future frames is then calculated using this model.

[0032] Combining an adversarial meta-learning strategy, the optimal energy threshold parameter in the current scene is set according to the environmental complexity factor. The optimal energy threshold parameter is compared with the hand energy cost function value. Based on the comparison results, the biomechanical prediction verification results of the gesture features in future frames are analyzed.

[0033] If the biomechanical prediction verification result of gesture features in future frames contains physical violations, then the violation gradient is calculated and the retrieval weight of the corresponding historical memory sample that caused the violation prediction is reduced.

[0034] Preferably, the step of performing topological matching based on a minimum spanning forest structure of candidate memories in the topological layer of the historical memory bank to select memory samples; and performing uncertainty-aware weighted fusion on the complete spatiotemporal features of the selected memory samples to generate preliminary completion features includes:

[0035] The three-dimensional coordinates of several key points are decoded from multi-scale spatiotemporal features, and the spatial position variance of each key point is output using Bayesian inference to form a Gaussian distribution probability model.

[0036] The probability-weighted Mahalanobis distance between all keypoint pairs is calculated based on the Gaussian distribution probability model, and the minimum spanning forest topology is constructed using the probability-weighted Mahalanobis distance.

[0037] In the topological layer of the historical memory bank, the topological similarity between the minimum spanning forest topological graph and the pre-stored minimum spanning forest structure of the candidate memory is calculated, and historical memory samples with similarity exceeding the threshold are selected to form a memory set with consistent structure.

[0038] For the complete spatiotemporal features corresponding to the memory set with consistent structure, confidence weighting is performed by combining the variance of key points, and the resulting memory context features are fused and used as prior features for preliminary completion, in order to identify and isolate limb fracture areas caused by severe occlusion.

[0039] Preferably, the construction of a hand biomechanical energy model including tendon tension and kinetic energy consumption, and the calculation of the hand energy cost function value of gesture features in future frames using the hand biomechanical energy model, includes:

[0040] Based on the bending angle, natural relaxation angle and physiological stiffness coefficient with several possible values ​​of the target joint in the gesture features of future frames, a Hooke's Law-like model is used to simulate the tensile potential energy of the hand tendon under extreme stretching.

[0041] Based on the finger bone moment of inertia and joint angular velocity of the target joint in the gesture features of future frames, the kinetic energy consumption during joint movement is simulated.

[0042] The energy cost function of the hand is obtained by summing the tensile potential energy of the tendons under extreme stretching and the kinetic energy consumption during joint movement.

[0043] Preferably, the step of fusing multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features, and scene depth perception features into a conditional vector, guiding the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area, and simultaneously generating repair features for the hand occlusion area includes:

[0044] A multi-source conditional encoder with a gated fusion network is used to extract multi-scale spatiotemporal features, preliminary completion features, and scene depth perception features containing hand occlusion information. At the same time, the multi-scale spatiotemporal features are used as the original observation features with occlusion, and the preliminary completion features are used as the prior features after verification by minimum generating forest. They are then uniformly projected to the same channel dimension for feature alignment.

[0045] The aligned features are concatenated along the channel dimension and then input into a two-layer perceptron with a sigmoid activation function at the end. The two-layer perceptron generates a gated fusion map.

[0046] The original features and prior features are complementary and weighted using a gated fusion graph to generate a conditional vector. The conditional vector is injected into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism. The symmetric encoder-decoder network predicts noise and gradually restores latent features. The latent features are used as the repair features of the hand occlusion area. The biomechanical prediction verification results and the repair features are used as the inputs to step S4.

[0047] Preferably, the step of injecting conditional vectors into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism, and predicting noise and progressively reconstructing latent features through the symmetric encoder-decoder network, includes:

[0048] The conditional vector is input into two independent mapping networks, and the affine transformation parameters, which include a scaling factor and a shift factor, are predicted simultaneously through the two mapping networks.

[0049] In the normalization layer of the symmetric encoder-decoder network, modulation operations are performed on the scale factor and offset factor to predict noise and gradually restore the latent features of the hand occlusion area.

[0050] Preferably, the expression for the condition vector is:

[0051] ;

[0052] In the formula, This indicates the original, occluded observation features. This represents the prior features retrieved from the memory and validated using a minimum spanning forest. It represents scene depth perception features and is used to provide occlusion relationships in three-dimensional space; Represents a condition vector; Indicates the probability distribution of the current region; Represents the balanced weighting coefficients of scene depth-aware features; symbol This indicates an element-wise multiplication operation.

[0053] According to another aspect of the present invention, an iterative diffusion gesture recognition system based on memory guidance and prediction verification is also provided, the system comprising:

[0054] The spatiotemporal feature extraction module is used to construct a multi-scale feature extraction network based on an improved deep residual network as the backbone network and based on the pre-defined convolutional kernel and stride parameters. The video with hand occlusion is input into the multi-scale feature extraction network, and the output is multi-scale spatiotemporal features containing hand occlusion information.

[0055] The memory-prediction reasoning module is used to perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features based on the historical memory bank, to obtain the generated preliminary completed features and future frame gesture features. It constructs a hand biomechanical energy model that includes tendon tension and kinetic energy consumption to physically verify the future frame gesture features and obtain biomechanical prediction verification results.

[0056] The conditional diffusion de-occlusion module is used to fuse multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features, and scene depth perception features into a conditional vector, which guides the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area and simultaneously generate repair features for the hand occlusion area.

[0057] The iteration control module is used to calculate the overall confidence level based on the repair features and biomechanical prediction verification results. If the overall confidence level does not reach the preset threshold, the historical memory bank is updated based on the intermediate results generated by the current iteration and returned to the memory-prediction inference module for a new round of prediction and diffusion process until the iteration conditions are met, then it stops and enters the classification output module. If the overall confidence level reaches the preset threshold, it directly enters the classification output module.

[0058] The classification output module is used to classify actions based on the final repaired features of the hand occlusion area after iteration, and output the gesture action classification results.

[0059] The beneficial effects of this invention are as follows:

[0060] 1. This invention is based on memory guidance of probabilistic graphical models, which breaks through the traditional Euclidean distance retrieval and constructs a minimum generating forest topology admission mechanism based on the propagation of key point uncertainty. By calculating the Bayesian probability weighted distance, it can automatically identify and isolate limb fracture areas caused by occlusion and prevent noisy samples from contaminating the memory bank.

[0061] 2. This invention is based on biomechanics and meta-learning prediction verification, which differs from traditional geometric threshold verification. By introducing a biomechanical energy model of the hand that includes tendon tension and kinetic energy consumption, and combining it with an adversarial meta-learning strategy, it can automatically search for the optimal physical verification parameters according to the complexity of the environment, thereby realizing adaptive kinematic control with physical interpretability.

[0062] 3. This invention designs a multi-source conditional encoder that includes an explicit gating fusion network. By automatically identifying occlusion areas and performing weighted complementarity through a learnable gating graph, combined with the dynamic modulation diffusion process of the feature space, it achieves precise decoupling and recombination of memory priors and observed features in a physical sense.

[0063] 4. Based on the closed-loop feedback and failure probability backpropagation mechanism, this invention uses a closed-loop iterative control with self-evolution capabilities to calculate the violation gradient and backpropagate it to the memory module when physical verification fails, automatically reducing the weight of historical memory that caused the erroneous prediction, thus realizing a dynamic optimization mechanism that learns from errors. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of an iterative diffusion gesture recognition method based on memory guidance and prediction verification according to an embodiment of the present invention;

[0066] Figure 2 This is a block diagram of the overall architecture of an iterative diffusion gesture recognition system based on memory guidance and prediction verification according to an embodiment of the present invention;

[0067] Figure 3This is a schematic diagram of the working principle of the conditional diffusion de-occlusion module in the iterative diffusion gesture recognition system based on memory guidance and prediction verification according to an embodiment of the present invention.

[0068] Figure 4 This is a memory-predictive inference module diagram in an iterative diffusion gesture recognition system based on memory guidance and predictive verification according to an embodiment of the present invention;

[0069] Figure 5 This is an iterative control logic diagram in an iterative diffusion gesture recognition system based on memory guidance and prediction verification according to an embodiment of the present invention;

[0070] Figure 6 This is a principle block diagram of an iterative diffusion gesture recognition system based on memory guidance and prediction verification according to an embodiment of the present invention.

[0071] In the picture:

[0072] 1. Spatiotemporal feature extraction module; 2. Memory-prediction reasoning module; 3. Conditional diffusion de-occlusion module; 4. Iteration control module; 5. Classification output module. Detailed Implementation

[0073] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention.

[0074] According to embodiments of the present invention, an iterative diffusion gesture recognition method and system based on memory guidance and prediction verification are provided.

[0075] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, an iterative diffusion gesture recognition method based on memory guidance and prediction verification includes:

[0076] S1. An improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on the convolution kernel and stride parameters of the preset specifications. The hand occlusion video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information are output.

[0077] S2. Based on the historical memory bank, perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features to obtain preliminary completed features and future frame gesture features. Construct a hand biomechanical energy model that includes tendon tension and kinetic energy consumption to physically verify the future frame gesture features and obtain biomechanical prediction verification results.

[0078] S3. Multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features and scene depth perception features are fused into a conditional vector to guide the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area and simultaneously generate repair features for the hand occlusion area.

[0079] S4. Calculate the overall confidence level based on the repair features and biomechanical prediction verification results; if the overall confidence level does not reach the preset threshold, update the historical memory bank based on the intermediate results generated in the current iteration and return to step S2 to carry out a new round of prediction and diffusion process until the iteration conditions are met and stop and proceed to step S5; if the overall confidence level reaches the preset threshold, proceed directly to step S5.

[0080] S5. After the iteration is completed, the action classification is performed based on the final repair features of the hand occlusion area, and the gesture action classification result is output.

[0081] In one embodiment, the improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on convolutional kernels and stride parameters of preset specifications. The hand-occluded video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information include:

[0082] Extract the RGB video sequence from the hand-occluded video and perform preprocessing;

[0083] The preprocessed RGB video sequence is input into the improved deep residual network. The first layer of the improved deep residual network performs a preliminary spatiotemporal convolution operation based on the preset three-dimensional convolution kernel and stride parameters to extract the initial spatiotemporal feature map.

[0084] The initial spatiotemporal feature map is input into the subsequent layers of the improved deep residual network. The subsequent layers of the improved deep residual network perform two-dimensional spatial convolution and one-dimensional temporal convolution operations in sequence based on spatial convolution kernels and temporal convolution kernels of preset specifications, and output spatiotemporal feature maps of different scales.

[0085] By fusing features from spatiotemporal feature maps of different scales along the time dimension, multi-scale spatiotemporal features containing hand occlusion information are obtained.

[0086] In one embodiment, the feature fusion of spatiotemporal feature maps at different scales along the time dimension to obtain multi-scale spatiotemporal features containing hand occlusion information includes:

[0087] Spatiotemporal feature maps of different scales are flattened along the time dimension into feature sequences, and corresponding query vectors, key vectors, and value vectors are generated for each element in the feature sequence.

[0088] The attention weight distribution is calculated based on the query vector and key vector, and the value vectors of all elements are weighted and summed according to the attention weight distribution to output temporal fusion feature maps at different scales.

[0089] Upsampling is performed on each temporal fusion feature map to adjust the spatial size to a uniform size; the temporal fusion feature maps with uniform spatial size are then stitched together along the channel dimension to generate multi-scale spatiotemporal features that include hand occlusion information.

[0090] In one embodiment, the initial gesture state prediction and feature completion operation based on the historical memory bank for multi-scale spatiotemporal features is performed to obtain preliminary completed features and future frame gesture features. A hand biomechanical energy model including tendon tension and kinetic energy consumption is constructed to physically verify the future frame gesture features, and the biomechanical prediction verification results include:

[0091] High-level semantic compression is performed on multi-scale spatiotemporal features, and approximate retrieval of the compressed feature vectors is performed in the semantic layer of the historical memory bank to obtain several candidate memories.

[0092] In the topological layer of the historical memory bank, topological matching is performed based on the minimum spanning forest structure of candidate memories to select memory samples; the complete spatiotemporal features of the selected memory samples are weighted and fused with uncertainty perception to generate preliminary completion features.

[0093] A bidirectional interactive Transformer decoder is constructed. The preliminary completed features are input into the bidirectional interactive Bayesian Transformer decoder to predict the probability distribution of gesture states in future frames and generate gesture features for future frames.

[0094] A biomechanical energy model of the hand, incorporating tendon tension and kinetic energy consumption, is constructed. The hand energy cost function value of gesture features in future frames is then calculated using this model.

[0095] Combining an adversarial meta-learning strategy, the optimal energy threshold parameter in the current scene is set according to the environmental complexity factor. The optimal energy threshold parameter is compared with the hand energy cost function value. Based on the comparison results, the biomechanical prediction verification results of the gesture features in future frames are analyzed.

[0096] If the biomechanical prediction verification result of gesture features in future frames contains physical violations, then the violation gradient is calculated and the retrieval weight of the corresponding historical memory sample that caused the violation prediction is reduced.

[0097] In one embodiment, the step of performing topological matching based on a minimum spanning forest structure of candidate memories in the topological layer of the historical memory bank to select memory samples, and performing uncertainty-aware weighted fusion on the complete spatiotemporal features of the selected memory samples to generate preliminary completion features includes:

[0098] The three-dimensional coordinates of several key points are decoded from multi-scale spatiotemporal features, and the spatial position variance of each key point is output using Bayesian inference to form a Gaussian distribution probability model.

[0099] The probability-weighted Mahalanobis distance between all keypoint pairs is calculated based on the Gaussian distribution probability model, and the minimum spanning forest topology is constructed using the probability-weighted Mahalanobis distance.

[0100] In the topological layer of the historical memory bank, the topological similarity between the minimum spanning forest topological graph and the pre-stored minimum spanning forest structure of the candidate memory is calculated, and historical memory samples with similarity exceeding the threshold are selected to form a memory set with consistent structure.

[0101] For complete spatiotemporal features corresponding to memory sets with consistent structure, confidence weighting is performed by combining keypoint variance, and memory context features are fused to generate memory context features as preliminary features for completion, so as to identify and isolate limb fracture areas caused by severe occlusion.

[0102] In one embodiment, constructing a hand biomechanical energy model that includes tendon tension and kinetic energy consumption, and using the hand biomechanical energy model to calculate the hand energy cost function value of gesture features in future frames, includes:

[0103] Based on the bending angle, natural relaxation angle and physiological stiffness coefficient with several possible values ​​of the target joint in the gesture features of future frames, a Hooke's Law-like model is used to simulate the tensile potential energy of the hand tendon under extreme stretching.

[0104] Based on the finger bone moment of inertia and joint angular velocity of the target joint in the gesture features of future frames, the kinetic energy consumption during joint movement is simulated.

[0105] The energy cost function of the hand is obtained by summing the tensile potential energy of the tendons under extreme stretching and the kinetic energy consumption during joint movement.

[0106] In one embodiment, the process of fusing multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features, and scene depth perception features into a conditional vector, guiding the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area, and simultaneously generating repair features for the hand occlusion area includes:

[0107] A multi-source conditional encoder with a gated fusion network is used to extract multi-scale spatiotemporal features, preliminary completion features, and scene depth perception features containing hand occlusion information. At the same time, the multi-scale spatiotemporal features are used as the original observation features with occlusion, and the preliminary completion features are used as the prior features after verification by minimum generating forest. They are then uniformly projected to the same channel dimension for feature alignment.

[0108] The aligned features are concatenated along the channel dimension and then input into a two-layer perceptron with a sigmoid activation function at the end. The two-layer perceptron generates a gated fusion map.

[0109] The original observed features and prior features are complementary and weighted using a gated fusion graph to generate a conditional vector. The conditional vector is injected into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism. The symmetric encoder-decoder network predicts noise and gradually restores latent features. The latent features are used as the repair features of the hand occlusion area. The biomechanical prediction verification results and the repair features are used as the inputs to step S4.

[0110] In one embodiment, the step of injecting conditional vectors into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism, and predicting noise and progressively reconstructing latent features through the symmetric encoder-decoder network, includes:

[0111] The conditional vector is input into two independent mapping networks, and the affine transformation parameters, which include a scaling factor and a shift factor, are predicted simultaneously through the two mapping networks.

[0112] In the normalization layer of the symmetric encoder-decoder network, modulation operations are performed on the scale factor and offset factor to predict noise and gradually restore the latent features of the hand occlusion area.

[0113] In one embodiment, the expression for the condition vector is:

[0114] ;

[0115] In the formula, This indicates the original, occluded observation features. This represents the prior features retrieved from the memory and validated using a minimum spanning forest. It represents scene depth perception features and is used to provide occlusion relationships in three-dimensional space; Represents a condition vector; Indicates the probability distribution of the current region; Represents the balanced weighting coefficients of scene depth-aware features; symbol This indicates an element-wise multiplication operation.

[0116] According to another embodiment of the invention, such as Figure 2 and Figure 6 As shown, an iterative diffusion gesture recognition system based on memory guidance and prediction verification is also provided. This system includes:

[0117] The spatiotemporal feature extraction module 1 is used to construct a multi-scale feature extraction network based on an improved deep residual network as the backbone network and based on the pre-defined convolutional kernel and stride parameters. The video of hand occlusion is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information are output.

[0118] The memory-prediction reasoning module 2 is used to perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features based on the historical memory bank, to obtain the generated preliminary completed features and future frame gesture features, and to construct a hand biomechanical energy model including tendon tension and kinetic energy consumption to physically verify the future frame gesture features, and obtain biomechanical prediction verification results.

[0119] Conditional diffusion de-occlusion module 3 is used to fuse multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features and scene depth perception features into a conditional vector, guide the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area, and simultaneously generate repair features for the hand occlusion area.

[0120] Iteration control module 4 is used to calculate the comprehensive confidence level based on the repair features and biomechanical prediction verification results. If the comprehensive confidence level does not reach the preset threshold, the historical memory bank is updated based on the intermediate results generated by the current iteration and returned to the memory-prediction inference module for a new round of prediction and diffusion process until the iteration conditions are met and then it stops and enters the classification output module 5. If the comprehensive confidence level reaches the preset threshold, it directly enters the classification output module 5.

[0121] Classification output module 5 is used to classify actions based on the final repair features of the hand occlusion area after iteration, and output the gesture action classification results.

[0122] To achieve robust gesture recognition in occluded scenarios, this invention provides an iterative diffusion gesture recognition method and system based on memory guidance and prediction verification. This invention constructs a complete iterative inference technology system by introducing an innovative memory-prediction mechanism and a conditional diffusion model. To facilitate understanding of the above technical solutions, the following further explains the technical solutions from the perspectives of architecture and principles:

[0123] As attached Figure 2 As shown, this invention comprises five core modules: a spatiotemporal feature extraction module 1, a memory-prediction inference module 2, a conditional diffusion de-occlusion module 3, an iterative control module 4, and a classification output module 5. Each module works collaboratively using a combination of feedforward and feedback mechanisms to form a dynamic iterative optimization loop. The workflow of this invention is as follows: Figure 1As shown: After inputting the occluded video, the spatiotemporal feature extraction module 1 extracts multi-scale spatiotemporal features. These features are input into the memory-prediction inference module 2, which performs initial gesture state prediction and feature completion based on historical information in the memory bank, and outputs the predicted future frame features for verification. On the other hand, these features, along with the conditional information output by the memory-prediction module 2, are input into the conditional diffusion de-occlusion module 3, which restores the clean features of the occluded area through a multi-step reverse diffusion process. The iterative control module 4 receives the repaired features and prediction verification results output by the conditional diffusion de-occlusion module 3, performs multi-dimensional confidence evaluation, and generates a feedback signal if the confidence level does not reach the threshold, triggering the system to return to the memory-prediction inference module 2, update the memory using the intermediate results of the current iteration, and perform a new round of prediction and diffusion reconstruction. After the iteration is completed, the classification output module 5 performs the final action classification and outputs the recognition result. This closed-loop mechanism of "extraction-memory prediction-diffusion repair-evaluation-iteration" is the key to the occlusion robustness of this system.

[0124] Among them, the spatiotemporal feature extraction module 1 extracts multi-scale spatiotemporal features, including:

[0125] ① Multi-scale feature extraction network;

[0126] The backbone network structure adopts an improved architecture based on 3D-ResNet34, replacing the original 2D convolution kernels with spatiotemporally separated (2+1)D convolution kernels, that is, first performing 2D spatial convolution and then performing 1D temporal convolution.

[0127] The input is an RGB video sequence, edited into 16 frames, with each frame scaled to 224×224 resolution.

[0128] The convolution kernels and strides are as follows: the first layer has a 7x7x7 convolution kernel (time x height x width) and a stride of 1x2x2; subsequent layers all use (2+1)D convolution, with a spatial convolution kernel of 3x3 and a temporal convolution kernel of 3.

[0129] The feature pyramid output includes four feature maps at different scales, with 64, 128, 256, and 512 channels, corresponding to time dimensions of 16, 8, 4, and 2, and spatial dimensions of 56x56, 28x28, 14x14, and 7x7.

[0130] The task is to classify gestures from complete video sequences by supervising pre-training on large gesture datasets NvGesture and EgoHands.

[0131] ② Temporal feature fusion mechanism;

[0132] The fusion structure involves independently applying a multi-head self-attention mechanism to fuse temporal information at each scale of the feature pyramid, specifically calculated as follows:

[0133] Flatten the feature map into a sequence along the time dimension; query ( ),key( ),value( The vector is obtained by linearly projecting the feature map, where the projection dimension is... The value is 256, based on the query ( ),key( And calculate attention weights along the projection dimension:

[0134] ;

[0135] Based on attention weights Features after fusion:

[0136] ;

[0137] After upsampling the fused features at each scale to 56x56, the channels are stitched together to form the final fused spatiotemporal features.

[0138] Among them, the memory-prediction reasoning module 2, when performing initial gesture state prediction and feature completion based on historical information in the memory bank, and outputting the predicted future frame features for verification, includes:

[0139] like Figure 4 As shown, the memory-prediction reasoning module 2 uses the historical memory bank to initially complete the current occlusion features and establishes a multi-scale prediction and adaptive motion verification mechanism to guide the subsequent iteration process.

[0140] (1) Constructing a probabilistic graph-guided hierarchical memory network and topology admission mechanism: In the design of the memory module, this invention breaks through the limitations of the traditional Euclidean distance metric and constructs a probabilistic graph model and hierarchical storage architecture that includes uncertainty propagation, specifically including:

[0141] ① Keypoint uncertainty propagation and Minimum Spanning Forest (MSF) construction

[0142] In the process of feature-to-skeleton conversion, this invention connects a fully connected regression head in parallel at the end of the multi-scale feature extraction network to map the multi-scale spatiotemporal features into the coordinate space of key points of the human hand skeleton, thereby outputting not only the three-dimensional coordinates of the key points. It also uses Bayesian inference to output the spatial location variance of each key point. This forms a Gaussian probability model.

[0143] Based on the above Gaussian distribution probability model, calculate the key points. and Probability-weighted Mahalanobis distance between As the weights of the edges in the topological graph. The construction process specifically includes: First, taking all hand keypoints as nodes, and then... First, construct a complete graph with edge weights; second, generate a minimum spanning tree (MST) using Kruskal's algorithm or Prim's algorithm; finally, set a break threshold. Traverse all edges in the MST and cut the weights. The edges (i.e., connections with excessive uncertainty or excessive distance) are used to split the MST into a Minimal Spanning Forest (MSF).

[0144] During the verification phase, the adjacency matrix of the current frame MSF is constructed and the MSF structure stored in the historical memory is matched with graph isomorphism or the graph edit distance is calculated. If the structural difference is less than the preset tolerance, it is considered to pass the topology verification. This step can effectively identify and isolate limb breakage or extremely distorted areas caused by severe occlusion, and prevent erroneous topology structures from being forcibly encoded.

[0145] This invention achieves initial gesture state prediction and feature completion through the following collaborative mechanism: First, it uses a semantic layer for fast retrieval to find candidate memories with similar content; second, it uses the MSF structure of the topological layer to filter candidates based on physical plausibility, eliminating samples that, although similar in features, violate pose rules; finally, it uses a double buffering mechanism to ensure that the samples stored in the memory bank are always high-quality data that has undergone rigorous biomechanical verification, and then weights and fuses the spatiotemporal features of the selected high-quality memory samples with the currently observed features to fill the feature gaps in the occluded areas, completing the initial completion. The specific architecture is as follows:

[0146] ② A hierarchical index architecture with semantic-topology decoupling:

[0147] The semantic layer (L1) stores compressed high-dimensional global semantic features and uses lightweight vector retrieval (e.g., the HNSW algorithm) for fast initial screening in milliseconds; the topology layer (L2) stores the corresponding MSF topology graph for fine-grained structure verification.

[0148] ③ Double-buffering mechanism for memory playback: To address the computational overhead caused by complex topology verification, this invention designs a double-buffering update strategy, which includes a buffer and a cleanup buffer:

[0149] The active buffer performs only lightweight checks for real-time inference reads; the cleanup buffer runs asynchronously in the background, using a high-precision MSF topology consistency algorithm to perform deep audits on newly written samples. If a sample is found to be semantically similar but has a slight topology drift that violates biomechanics, it is removed from the active buffer. This mechanism ensures the self-cleaning capability of the memory bank during long-term operation.

[0150] (2) Based on biomechanical constraints, Bayesian bidirectional prediction and meta-learning verification: This module introduces deep biomechanical constraints and meta-learning strategies on the basis of the original kinematic velocity verification to achieve more physically interpretable verification.

[0151] ① Bidirectional interactive Bayesian Transformer decoder:

[0152] The decoder's output layer is replaced with a Bayesian neural network (BNN) layer to achieve Bayesian interval prediction. For each predicted future joint position, the network outputs not just a single coordinate point, but a probability distribution. Based on this, a 95% "Credible Interval" is generated. If the observed gesture features fall outside this credible interval, they will be directly judged as an anomaly without relying on a hard threshold.

[0153] ②Adversarial meta-learning verification based on biomechanical energy consumption:

[0154] This invention abandons the single angular velocity threshold and instead constructs a hand energy cost function as a biomechanical energy model: ;

[0155] in, (Tension potential energy) is used to simulate the elastic potential energy of hand tendons under extreme tension, specifically calculated using a Hooke's Law-like model:

[0156]

[0157] In the formula, The number of hand joints (in this embodiment, N=21 is taken based on the standard hand skeleton model); For the currently predicted number The bending angle of each joint; This is the resting angle of the joint, that is, the natural angle when the hand muscles are completely relaxed (usually set to 100°). (anatomical statistical mean) For the first The physiological stiffness coefficients corresponding to each joint reflect the physical differences in the bending resistance of different joints, such as the base of the thumb and the tip of the little finger. It is the tension potential energy.

[0158] It should be noted that, The methods for obtaining values ​​include two strategies:

[0159] The first strategy is an anatomically pre-defined model, based on medical statistical data, such as ergonomic databases, which will... Set as a fixed constant that reflects the characteristics of real human tendons;

[0160] The second strategy is an adaptive learning mode, which will Defined as learnable parameters in a neural network, these parameters are automatically adjusted by the model through backpropagation during training. The value of allows it to dynamically adapt to different users, such as different hand sizes and stiffness, personalized movement characteristics. This design enables the biomechanical model to not only have theoretical constraints, but also data-driven generalization capabilities.

[0161] (Kinetic energy) is used to simulate the energy consumption during rapid joint movement. The specific calculation formula is as follows:

[0162]

[0163] In the formula, For the moment of inertia of the finger bones, The joint angular velocity, It is kinetic energy.

[0164] Specifically, according to human physiology, the human hand always tends to move along the path of least energy. If we predict hand gestures... An abnormal surge indicates that the action violates anatomical norms, such as excessive bending of the fingers in the opposite direction.

[0165] Adversarial Meta-Learning: Traditional energy thresholds are difficult to set. This invention introduces a meta-learner, which takes the current environmental complexity factor as an example. As input, the output is the optimal energy threshold parameter for the current scenario. The adversarial meta-learning training strategy employs an adversarial training approach. The generator attempts to generate samples that are extremely close to reality but have minor physical violations, while the meta-learner tries to find the optimal threshold parameters that can accurately intercept these samples.

[0166] ③ When the verification module determines that the current prediction is physically illegal, it calculates the gradient of the degree of illegality. This gradient is not only used to correct the current frame, but also backpropagated to the memory retrieval module. It also reduces the retrieval weight of the "historical memory key-value pair" that caused the erroneous prediction. This mechanism enables the system to learn from the erroneous prediction and automatically suppress those poor memory samples that are prone to causing unnatural predictions.

[0167] Among them, the conditional diffusion demasking module 3 includes the following steps when restoring the cleanliness characteristics of the occluded area through a multi-step reverse diffusion process:

[0168] like Figure 3As shown, this module is in the core loop of the system and is responsible for gradually removing occlusion noise by using multi-source conditional information. Its core innovation lies in the introduction of scene depth perception and feature space dynamic modulation mechanism.

[0169] (1) Multi-source conditional coding and gating fusion network;

[0170] ① The multi-source input is that the encoder receives three feature inputs, specifically including: the current occluded original observation features (dimensions). ) Prior features retrieved from the memory and verified by MST. Scene depth perception features are used to provide occlusion relationships in 3D space. .

[0171] ②The specific structure of the gated fusion network;

[0172] To avoid feature aliasing caused by direct splicing, this invention designs a pixel-level adaptive gating unit, and the specific processing steps are as follows:

[0173] Step A, , , pass Convolutional layers are uniformly projected to the same channel dimension. .

[0174] Step B: Concatenate the aligned three-channel features along the channel dimension, input them into a two-layer perceptron MLP, and connect a sigmoid activation function at the end to generate a gated fusion map. :

[0175] ;

[0176] In the formula, , The weight matrix is ​​a learnable matrix. It is the ReLU activation function. The value represents the probability distribution of the model's determination that the current region is a foreground hand, tending towards 1, or a background occlusion, tending towards 0.

[0177] Step C: Use a gating graph to perform complementary weighting on the original features and prior features to generate the final conditional vector. :

[0178] ;

[0179] In the formula, This indicates the original, occluded observation features. This represents the prior features retrieved from the memory and validated using a minimum spanning forest. It represents scene depth perception features and is used to provide occlusion relationships in three-dimensional space; Represents a condition vector; Represents the probability distribution of the current region, where, The balancing weight coefficients represent the depth-aware features of the scene, used to adjust the degree of influence of depth information on the final conditional vector; (symbol) This indicates an element-wise multiplication operation.

[0180] ③ Implementation of Feature Space Dynamic Modulation (FSDM);

[0181] get Then, through two independent mapping networks and (Each layer consists of 3 fully connected layers), predicting affine transformation parameters: scale factor. and offset factor Subsequently, modulation operations are performed at the normalization layer of the U-Net, thereby injecting multi-source fusion information into the diffusion generation process. The expression for performing modulation operations at the normalization layer of the U-Net is:

[0182] ;

[0183] In the formula, This represents the intermediate feature map input to the normalization layer; and They represent Mean and standard deviation along the channel dimension; and They represent the condition vectors respectively. The scale factor and shift factor, predicted by the mapping network, are used to modulate the features using affine transformation. This indicates the output characteristics after dynamic modulation.

[0184] (2) Implementation steps of multi-step diffusion reasoning;

[0185] Sampling pure noise ; Using the FSDM mechanism to condition Inject U-Net, predict noise, and progressively restore latent features according to the DDPM inverse denoising formula. The specific inverse denoising iterative formula is as follows:

[0186]

[0187] In the formula, For the first Potential characteristics of the step For the denoised first Step characteristics; These are the preset noise scheduling parameters. for The cumulative terms; The noise component predicted by the network; It is Gaussian random noise; This is the variance term. It is processed using the inverse denoising iterative formula. Through iterative steps, the potential features of the hand-occluded area were finally reconstructed. .

[0188] The iterative control module 4 receives the repaired features and prediction verification results output by the conditional diffusion demasking module 3, performs multi-dimensional confidence evaluation, and generates a feedback signal if the confidence level does not reach the threshold, triggering the return-to-memory-prediction inference module 2. This module updates the memory using the intermediate results of the current iteration and performs a new round of prediction and diffusion reconstruction, including:

[0189] like Figure 5 As shown, this module serves as both a discriminator and a controller, determining whether the inference process terminates or proceeds to the next round of optimization.

[0190] ① Overall confidence level assessment:

[0191] Calculate the overall confidence level: ,in, and These are weighted balance coefficients for classification self-consistency probability and memory similarity, used to measure the credibility of the current inference result; For classification self-consistency probability, To achieve cosine similarity with historical memory.

[0192] ② Dual termination logic and closed-loop feedback: In order to achieve a balance between accuracy and efficiency, this invention adopts a decision logic that combines absolute threshold and differential convergence for dual cancellation decision.

[0193] Differential convergence terminates: if However, the feature differences between two adjacent iterations (Minimum threshold) indicates that the model has converged or is oscillating, forcibly terminating the iteration and outputting the current best result. Among these, Indicates the current number The latent hand features reconstructed during round-by-round iterative inference Indicates in the previous round (i.e., the 1st round) The latent hand features reconstructed during iterative inference (round) are used. The comprehensive confidence calculation formula is used to calculate the magnitude of feature change during continuous iterations; when the magnitude of change is less than a threshold... When the feature is stable, it is considered that there is no need to continue consuming computing resources.

[0194] If none of the above conditions are met and the maximum number of attempts has not been reached, a feedback loop is triggered, wherein the execution of the feedback loop includes:

[0195] Memory update: After verifying the current reconstructed feature x0 with MST, attempt to write it into the memory.

[0196] Conditional update: Regenerate prior features using the updated memory. and conditions .

[0197] Further reasoning, the diffusion model in A new round of noise reduction will be carried out under guidance.

[0198] After the iteration is completed, the classification output module 5 performs the final action classification and outputs the recognition result. The training strategy technology scheme includes:

[0199] The first stage is module pre-training, in which the spatiotemporal feature extractor, diffusion model, and memory module are pre-trained on the complete dataset, synthetic occlusion data pairs, and continuous gesture sequences, respectively.

[0200] The second stage is end-to-end joint training using a course learning strategy. In the initial stage, slightly occluded samples are used with a fixed 1 iteration; in the middle stage, moderately occluded samples are used with 2-3 iterations allowed; and in the later stage, severely occluded samples are used with a maximum of 5 iterations.

[0201] Among them, the multi-task loss function is the consistency loss for visibility perception. In order to balance the preservation of known information and the reasonable inference of unknown regions, this invention designs a loss function with dynamic weights:

[0202] ;

[0203] In the formula, These represent the classification loss weight, diffusion reconstruction loss weight, consistency constraint loss weight, and memory retrieval loss weight, respectively, used to balance the contribution ratio of each task module in joint training.

[0204] Among them, the core Use visibility mask Apply region-specific constraints:

[0205] ;

[0206] In the formula, This is a visibility mask, with a value of 1 for unmasked areas and a value of 0 for masked areas; This represents the reconstructed hand features or the decoded image. Ground Truth labels representing true hand features. These are the weighting coefficients for the kinematic constraint loss; This is a kinematic penalty based on bone length consistency and joint angle limitations.

[0207] Visible area (M) vis =1) Use L1 loss to force the model to accurately recover known pixels, ensuring data fidelity.

[0208] Obstructed area ( =0) Relax pixel alignment and instead apply kinematic constraint loss to penalize predictions that violate the limits of bone length and joint angle, guiding the model to generate reasonable inferences that conform to physiological common sense.

[0209] To verify the effectiveness of the proposed method, comprehensive comparative and ablation experiments were conducted. The experiments were performed on the publicly available dataset EgoHands and the self-built complex occlusion gesture dataset Occlusion-Gesture, with the experimental setup including:

[0210] ① Dataset;

[0211] The EgoHands dataset contains 4,800 video samples of 48 types of gestures. From these, 2,000 samples with varying degrees of occlusion were selected as the test set.

[0212] Occlusion-Gesture is a self-built dataset containing 20 categories of everyday interactive gestures and a total of 5,000 video samples. It is specifically designed to simulate complex scenarios such as static object occlusion, dynamic self-occlusion, and multi-handed interaction.

[0213] The evaluation metrics are Top-1 accuracy (%), F1 score (%), and average processing latency (ms).

[0214] ② Baseline method;

[0215] Baseline 1: 3D-CNN;

[0216] Baseline 2: MediaPipe Hands+LSTM (a classifier based on keypoint sequences);

[0217] Baseline 3: ST-GCN (Spatiotemporal Graph Convolutional Network);

[0218] Baseline 4: GAN-Based Inpainting (based on the repair method of US20210110615A1, adapted for this task).

[0219] ③ Experimental environment: Ubuntu 18.04, NVIDIA RTX 3080 GPU, Intel i9-10900K CPU.

[0220] The overall performance comparison experiment shows that the overall performance of this method and the baseline methods on the mixed test set is shown in Table 1.

[0221] Table 1. Overall performance comparison of our method with baseline methods on the mixed test set.

[0222] Existing methods Top-1 accuracy (%) F1 score (%) Average latency (ms) 3D-CNN (Baseline 1) 68.5 65.2 25.1 MediaPipe+LSTM (Baseline 2) 72.3 70.1 33.5 ST-GCN (Baseline 3) 75.6 73.8 28.7 GAN-Based (Baseline 4) 77.4 75.9 41.3 Method of the present invention 92.3 91.5 31.8

[0223] As shown in Table 1, the method of the present invention significantly outperforms all baseline methods in terms of recognition accuracy and F1 score, demonstrating its comprehensive advantages in complex occlusion scenarios. Meanwhile, the processing speed meets the real-time requirements (>30fps).

[0224] Robustness tests under different degrees of occlusion were conducted. The test samples were divided into three groups according to the proportion of the visible area of ​​the hand. The comparison results are shown in Table 2.

[0225] Table 2 Robustness test under different occlusion levels

[0226] Existing methods Slight occlusion (visible >70%) Moderate shading (30%-70% visibility) Severe occlusion (visible <30%) 3D-CNN (Baseline 1) 89.5% 65.3% 32.7% MediaPipe+LSTM (Baseline 2) 93.1% 70.2% 41.5% ST-GCN (Baseline 3) 94.2% 72.8% 48.9% GAN-Based (Baseline 4) 94.8% 76.1% 55.6% Method of the present invention 95.5% 91.8% 85.4%

[0227] As shown in Table 2, all methods perform similarly under slight occlusion, but as occlusion intensifies, the performance of the baseline methods drops sharply. However, the method of this invention, with its powerful feature reasoning and completion capabilities, can still maintain a high accuracy of 85.4% under severe occlusion, demonstrating extremely strong robustness.

[0228] Ablation experiments were conducted to verify the contributions of each core module. The results are shown in Table 3, where w / o indicates no ablation.

[0229] Table 3. Verification of the contributions of each core module through ablation experiments.

[0230] Experimental setup Top-1 accuracy (%) relative decline Complete model 92.3 - w / o memory module (remove historical information) 83.1 -9.2 w / o conditional diffusion (replaced by linear interpolation) 79.5 -12.8 w / o iterative control (fixed to 1 inference) 85.7 -6.6 w / o prediction verification 88.9 -3.4

[0231] As shown in Table 3, the memory module is the component that contributes the most to the performance. Its absence leads to a 9.2% decrease in accuracy, demonstrating the importance of utilizing historical information. The conditional diffusion module is crucial to the quality of feature recovery. Its absence leads to the largest performance decrease of 12.8%. The iterative control mechanism can effectively improve the accuracy by about 6.6%, demonstrating the value of dynamically optimizing the inference depth. Although the prediction verification module contributes relatively little, it can further improve the reliability of the system.

[0232] The relationship between iteration count and performance / efficiency was analyzed. The average number of iterations required for the model to reach the termination condition when processing the test set and the corresponding performance are shown in Table 4.

[0233] Table 4. Average number of iterations required to reach the termination condition and their corresponding performance.

[0234] Number of iterations Sample proportion Average accuracy of this group Average cumulative latency (ms) 1 25.3% 96.8% 18.5 2 41.5% 94.1% 26.3 3 22.7% 90.5% 35.2 4 7.2% 86.3% 43.8 5 3.3% 82.9% 52.1

[0235] As shown in Table 4, approximately 66.8% of the samples can be identified with high accuracy within two iterations, demonstrating the system's high efficiency. For the most complex 3.3% of samples, the system can still provide valuable identification results through a maximum of five iterations, proving its strong ability to tackle challenging problems. The iterative control mechanism successfully achieves an adaptive balance between accuracy and efficiency.

[0236] The above experimental data fully demonstrates that the iterative diffusion gesture recognition method and system based on memory guidance and prediction verification provided by this invention significantly outperforms existing technologies in terms of recognition accuracy, occlusion robustness, spatiotemporal consistency, and real-time performance, achieving the expected technical effects. This invention effectively overcomes the performance bottleneck of existing technologies under extreme occlusion conditions. Compared with existing technologies, the recognition accuracy is significantly improved in severely occluded scenarios. The specific effects are reflected in the following aspects:

[0237] 1) Overcoming occlusion limitations to achieve high-precision recognition: Even in severely occluded scenarios with a visible area of ​​less than 30%, the gesture recognition accuracy can still reach 92.3%, which is more than 35% higher than traditional methods.

[0238] 2) It has intelligent iterative optimization capabilities. Through the memory-prediction iterative reasoning mechanism, the system can dynamically adjust the reasoning depth according to the current recognition confidence, thereby optimizing computational efficiency while ensuring accuracy.

[0239] 3) Maintain spatiotemporal continuity and use a conditional diffusion model for feature reconstruction to ensure a smooth transition of the repair results in the time series and a reasonable spatial structure.

[0240] 4) Achieve real-time processing performance, reaching a processing speed of over 30fps on standard GPU hardware to meet the real-time requirements of human-computer interaction.

[0241] 5) It has strong generalization ability and can adapt to various complex occlusion scenarios, including static object occlusion, dynamic self-occlusion, multi-hand interaction occlusion and other complex situations.

[0242] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An iterative diffusion gesture recognition method based on memory guidance and prediction verification, characterized in that, The method includes: S1. An improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on the convolution kernel and stride parameters of the preset specifications. The hand occlusion video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information are output. S2. Based on the historical memory bank, perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features to obtain preliminary completed features and future frame gesture features. Construct a hand biomechanical energy model that includes tendon tension and kinetic energy consumption to physically verify the future frame gesture features and obtain biomechanical prediction verification results. S3. Multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features and scene depth perception features are fused into a conditional vector to guide the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area and simultaneously generate repair features for the hand occlusion area. S4. Calculate the overall confidence level based on the repair features and biomechanical prediction verification results; if the overall confidence level does not reach the preset threshold, update the historical memory bank based on the intermediate results generated in the current iteration and return to step S2 to carry out a new round of prediction and diffusion process until the iteration conditions are met and stop and proceed to step S5; if the overall confidence level reaches the preset threshold, proceed directly to step S5. S5. After the iteration is completed, the action classification is performed based on the final repair features of the hand occlusion area, and the gesture action classification result is output.

2. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 1, characterized in that, The improved deep residual network is used as the backbone network, and a multi-scale feature extraction network is constructed based on the pre-defined convolutional kernel and stride parameters. The hand occlusion video is input into the multi-scale feature extraction network, and the output multi-scale spatiotemporal features containing hand occlusion information include: Extract the RGB video sequence from the hand-occluded video and perform preprocessing; The preprocessed RGB video sequence is input into the improved deep residual network. The first layer of the improved deep residual network performs a preliminary spatiotemporal convolution operation based on a three-dimensional convolution kernel and stride parameters of preset specifications to extract the initial spatiotemporal feature map. The initial spatiotemporal feature map is input into the subsequent layers of the improved deep residual network. The subsequent layers of the improved deep residual network perform two-dimensional spatial convolution and one-dimensional temporal convolution operations in sequence based on spatial convolution kernels and temporal convolution kernels of preset specifications, and output spatiotemporal feature maps of different scales. By fusing features from spatiotemporal feature maps of different scales along the time dimension, multi-scale spatiotemporal features containing hand occlusion information are obtained.

3. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 2, characterized in that, The process of fusing features from spatiotemporal feature maps of different scales along the time dimension to obtain multi-scale spatiotemporal features containing hand occlusion information includes: Spatiotemporal feature maps of different scales are flattened along the time dimension into feature sequences, and corresponding query vectors, key vectors, and value vectors are generated for each element in the feature sequence. The attention weight distribution is calculated based on the query vector and key vector, and the value vectors of all elements are weighted and summed according to the attention weight distribution to output temporal fusion feature maps at different scales. Upsampling is performed on each temporal fusion feature map to adjust the spatial size to a uniform size; the temporal fusion feature maps with uniform spatial size are then stitched together along the channel dimension to generate multi-scale spatiotemporal features that include hand occlusion information.

4. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 1, characterized in that, The process involves performing initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features based on a historical memory bank, resulting in preliminary completed features and future frame gesture features. A biomechanical energy model of the hand, incorporating tendon tension and kinetic energy consumption, is then constructed to physically verify the future frame gesture features. The resulting biomechanical prediction verification results include: High-level semantic compression is performed on multi-scale spatiotemporal features, and approximate retrieval of the compressed feature vectors is performed in the semantic layer of the historical memory bank to obtain several candidate memories. In the topological layer of the historical memory bank, topological matching is performed based on the minimum spanning forest structure of candidate memories to select memory samples; the complete spatiotemporal features of the selected memory samples are weighted and fused with uncertainty perception to generate preliminary completion features. A bidirectional interactive Transformer decoder is constructed. The preliminary completed features are input into the bidirectional interactive Bayesian Transformer decoder to predict the probability distribution of gesture states in future frames and generate gesture features for future frames. A biomechanical energy model of the hand, incorporating tendon tension and kinetic energy consumption, is constructed. The hand energy cost function value of gesture features in future frames is then calculated using this model. Combining an adversarial meta-learning strategy, the optimal energy threshold parameter in the current scene is set according to the environmental complexity factor. The optimal energy threshold parameter is compared with the hand energy cost function value. Based on the comparison results, the biomechanical prediction verification results of the gesture features in future frames are analyzed. If the biomechanical prediction verification result of gesture features in future frames contains physical violations, then the violation gradient is calculated and the retrieval weight of the corresponding historical memory sample that caused the violation prediction is reduced.

5. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 4, characterized in that, The memory samples are selected by performing topological matching based on the minimum spanning forest structure of candidate memories in the topological layer of the historical memory bank. Uncertainty-aware weighted fusion is performed on the complete spatiotemporal features of the selected memory samples to generate preliminary completion features, including: The three-dimensional coordinates of several key points are decoded from multi-scale spatiotemporal features, and the spatial position variance of each key point is output using Bayesian inference to form a Gaussian distribution probability model. The probability-weighted Mahalanobis distance between all keypoint pairs is calculated based on the Gaussian distribution probability model, and the minimum spanning forest topology is constructed using the probability-weighted Mahalanobis distance. In the topological layer of the historical memory bank, the topological similarity between the minimum spanning forest topological graph and the pre-stored minimum spanning forest structure of the candidate memory is calculated, and historical memory samples with similarity exceeding the threshold are selected to form a memory set with consistent structure. For complete spatiotemporal features corresponding to memory sets with consistent structure, confidence weighting is performed by combining keypoint variance, and memory context features are fused to generate memory context features as preliminary features for completion, so as to identify and isolate limb fracture areas caused by severe occlusion.

6. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 4, characterized in that, The construction of a hand biomechanical energy model that includes tendon tension and kinetic energy consumption, and the calculation of the hand energy cost function value of gesture features in future frames using the hand biomechanical energy model, include: Based on the bending angle, natural relaxation angle and physiological stiffness coefficient with several possible values ​​of the target joint in the gesture features of future frames, a Hooke's Law-like model is used to simulate the tensile potential energy of the hand tendon under extreme stretching. Based on the finger bone moment of inertia and joint angular velocity of the target joint in the gesture features of future frames, the kinetic energy consumption during joint movement is simulated. The energy cost function of the hand is obtained by summing the tensile potential energy of the tendons under extreme stretching and the kinetic energy consumption during joint movement.

7. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 1, characterized in that, The process of fusing multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features, and scene depth perception features into a conditional vector, guiding the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area, and simultaneously generating repair features for the hand occlusion area includes: A multi-source conditional encoder with a gated fusion network is used to extract multi-scale spatiotemporal features, preliminary completion features, and scene depth perception features containing hand occlusion information. At the same time, the multi-scale spatiotemporal features are used as the original observation features with occlusion, and the preliminary completion features are used as the prior features after verification by minimum generating forest. They are then uniformly projected to the same channel dimension for feature alignment. The aligned features are concatenated along the channel dimension and then input into a two-layer perceptron with a sigmoid activation function at the end. The two-layer perceptron generates a gated fusion map. The original observed features and prior features are complementary and weighted using a gated fusion graph to generate a conditional vector. The conditional vector is injected into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism. The symmetric encoder-decoder network predicts noise and gradually restores latent features. The latent features are used as the repair features of the hand occlusion area. The biomechanical prediction verification results and the repair features are used as the inputs to step S4.

8. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 7, characterized in that, The step of injecting conditional vectors into a symmetric encoder-decoder network with skip connections using a feature space dynamic modulation mechanism, and then predicting noise and gradually restoring latent features through the symmetric encoder-decoder network, includes: The conditional vector is input into two independent mapping networks, and the affine transformation parameters, which include a scaling factor and a shift factor, are predicted simultaneously through the two mapping networks. In the normalization layer of the symmetric encoder-decoder network, modulation operations are performed on the scale factor and offset factor to predict noise and gradually restore the latent features of the hand occlusion area.

9. The iterative diffusion gesture recognition method based on memory guidance and prediction verification according to claim 8, characterized in that, The expression for the condition vector is: ； In the formula, This indicates the original, occluded observation features. This represents the prior features retrieved from the memory and validated using a minimum spanning forest. It represents scene depth perception features and is used to provide occlusion relationships in three-dimensional space; Represents a condition vector; Indicates the probability distribution of the current region; Represents the balanced weighting coefficients of scene depth-aware features; symbol This indicates an element-wise multiplication operation.

10. An iterative diffusion gesture recognition system based on memory guidance and prediction verification, used to implement the iterative diffusion gesture recognition method based on memory guidance and prediction verification as described in any one of claims 1-9, characterized in that, The system includes: The spatiotemporal feature extraction module is used to construct a multi-scale feature extraction network based on an improved deep residual network as the backbone network and convolution kernels and stride parameters of preset specifications. The video with hand occlusion is input into the multi-scale feature extraction network, and the output is multi-scale spatiotemporal features containing hand occlusion information. The memory-prediction reasoning module is used to perform initial gesture state prediction and feature completion operations on multi-scale spatiotemporal features based on the historical memory bank, to obtain the generated preliminary completed features and future frame gesture features. It constructs a hand biomechanical energy model that includes tendon tension and kinetic energy consumption to physically verify the future frame gesture features and obtain biomechanical prediction verification results. The conditional diffusion de-occlusion module is used to fuse multi-scale spatiotemporal features containing hand occlusion information, preliminary completion features, and scene depth perception features into a conditional vector, which guides the diffusion probability model to perform multi-step reverse denoising on the hand occlusion area and simultaneously generate repair features for the hand occlusion area. The iteration control module is used to calculate the overall confidence level based on the repair features and biomechanical prediction verification results. If the overall confidence level does not reach the preset threshold, the historical memory bank is updated based on the intermediate results generated by the current iteration and returned to the memory-prediction inference module for a new round of prediction and diffusion process until the iteration conditions are met, then it stops and enters the classification output module. If the overall confidence level reaches the preset threshold, it directly enters the classification output module. The classification output module is used to classify actions based on the final repaired features of the hand occlusion area after iteration, and output the gesture action classification results.

Images