A dual-path parallel motion data optimization method fusing spatio-temporal prior and attention

Abstract

The application belongs to the technical field of motion data optimization, and discloses a dual-path parallel motion data optimization method fusing space-time prior and attention. The method proposes a dual-prior attention module, fuses space and time prior constraints, is beneficial to improving the modeling capability of a joint structure and time sequence dependence of a Transformer, and strengthens the spatial rationality of a skeleton structure. An affinity matrix is constructed through a skeleton connection matrix and a learnable topological matrix, spatial rationality and time sequence continuity are enhanced, and prior bias is superimposed in attention calculation, so that data driving and kinematic constraints are considered, and the robustness and accuracy of the model are improved. Compared with a traditional method, the three-dimensional motion data optimized by the method is more accurate in detail processing, and in particular, the accuracy is obviously improved in the expression of fast gestures or small actions. In addition, the model structure is simplified, and the calculation efficiency is also obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of motion data optimization technology, specifically involving a dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention, which is particularly suitable for scenarios such as sports science analysis, augmented reality systems, and medical rehabilitation monitoring. Background Technology

[0002] Human pose estimation (HPE), a fundamental task in computer vision, aims to extract the spatial coordinates of human joints from input images or video sequences. This technology has wide applications in various fields, such as sports science analysis, augmented reality (AR) systems, and medical rehabilitation monitoring. For example, in sports science, HPE technology provides analytical tools for athlete training. In badminton training, by combining traditional teaching methods with HPE-based quantitative analysis, coaches can overcome the limitations of traditional subjective evaluation and achieve data-driven precision teaching. In medical rehabilitation, HPE technology, by analyzing patients' motion videos, can effectively assist in the diagnosis of neurodegenerative diseases. Particularly for stroke rehabilitation training, the system can generate skeletal representations of the patient's posture. This motion data provides important references for monitoring the rehabilitation process and optimizing treatment plans.

[0003] In the field of computer vision, human pose estimation faces many challenges, with occlusion being particularly prominent. Occlusion mainly falls into two categories: environmental occlusion, caused by other objects in the scene, and self-occlusion, where parts of the human body themselves occlude each other. These occlusions lead to the loss of keypoint information, severely impacting the accuracy of pose estimation. Currently, the main motion data optimization methods for addressing occlusion in human pose estimation include data augmentation-based preprocessing methods, feature-discriminative architecture design methods, and deep learning-based optimization methods. Data augmentation methods aim to improve the robustness of the model by increasing the number of occluded samples in the training data. While traditional data augmentation methods (such as rotation and scaling) can handle simple image transformations, their effectiveness is limited when dealing with occlusion problems. Feature-discriminative architecture design methods, at the feature extraction level, enhance the model's ability to distinguish occluded features by improving the network structure.

[0004] With the widespread application of deep learning technology, especially Convolutional Neural Networks (CNNs), the HPE technology has made significant progress. However, current deep learning methods still face several key challenges: (1) Existing methods are insufficient in terms of image feature utilization, the generalization ability of the model needs to be improved, and there is a lack of sufficient mining of the temporal features of video sequences; (2) Occlusion problems in complex scenes (such as self-occlusion during motion) and changes in lighting conditions still cause problems for the robustness of the system; (3) The accuracy and smoothness of pose estimation for fine parts such as hands still need to be further improved to meet the needs of complex motion analysis. Therefore, motion data optimization is of great significance for the denoising of human pose estimation. Hand pose estimation faces unique challenges: high degree of freedom, severe self-occlusion and strong self-similarity. Summary of the Invention

[0005] The purpose of this invention is to propose a dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention. By proposing a 3D skeleton keypoint optimization framework that integrates spatial and temporal prior knowledge, spatial prior attention and temporal prior attention are embedded in the Transformer architecture. The optimization uses prior knowledge of human motion to solve the occlusion problem. Motion data is inherently spatiotemporally correlated, which provides a theoretical basis for denoising and optimization. The prior knowledge-based method mainly utilizes two key characteristics, namely the spatial dependence between joints and the temporal continuity of motion, to improve the smoothness of 3D motion data.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A dual-path parallel motion data optimization method integrating spatiotemporal priors and attention includes the following steps:

[0008] Step 1. Obtain the 3D motion data sequence, and use the 3D coordinates of each joint contained in each frame of 3D motion data in the 3D motion data sequence as the 3D motion data to be optimized, and construct the training dataset;

[0009] Step 2. Build a data optimization model that integrates spatiotemporal prior attention, which includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head;

[0010] First, the three-dimensional motion data to be optimized is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively.

[0011] The high-dimensional input features containing kinematic and temporal prior knowledge are added with positional and temporal codes respectively, and the high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are linearly fused with spatiotemporal priors.

[0012] Subsequently, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to further enhance the interaction and dependence between spatial and temporal features;

[0013] Then, the output features of the spatial multi-head attention module and the temporal multi-head attention module are linearly fused through spatiotemporal multi-head attention and input into the regression head for mapping, ultimately obtaining optimized and smooth 3D motion data output;

[0014] Step 3. Train the model using the training dataset, and then use the trained model to optimize motion data.

[0015] Furthermore, this invention also proposes a dual-path parallel motion data optimization system that integrates spatiotemporal priors and attention, corresponding to the above method, which adopts the following technical solution:

[0016] A dual-path parallel motion data optimization system integrating spatiotemporal priors and attention includes the following modules:

[0017] The preprocessing module is used to acquire the three-dimensional motion data sequence, take the three-dimensional coordinates of each joint contained in each frame of the three-dimensional motion data sequence as the three-dimensional motion data to be optimized, and construct the training dataset.

[0018] And a data optimization module, used to build a data optimization model that integrates spatiotemporal prior attention, which includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head;

[0019] First, the three-dimensional motion data to be optimized is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively.

[0020] The high-dimensional input features containing kinematic and temporal prior knowledge are added with positional and temporal codes respectively, and the high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are linearly fused with spatiotemporal priors.

[0021] Subsequently, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to further enhance the interaction and dependence between spatial and temporal features;

[0022] Then, the output features of the spatial multi-head attention module and the temporal multi-head attention module are linearly fused through spatiotemporal multi-head attention and input into the regression head for mapping, ultimately obtaining optimized and smooth 3D motion data output;

[0023] The model is trained using the training dataset, and the trained model is then used to optimize motion data.

[0024] Furthermore, the present invention also proposes a computer device comprising a memory and one or more processors. The memory stores executable code, which, when executed by the processors, is used to implement the aforementioned dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention.

[0025] Furthermore, this invention proposes a computer-readable storage medium storing a program thereon. When executed by a processor, this program is used to implement the aforementioned dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention.

[0026] The present invention has the following advantages:

[0027] As described above, this invention discloses a dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention. This method proposes a dual-priority attention module, fusing spatial and temporal prior constraints to enhance the Transformer's modeling ability for joint structures and temporal dependencies, thereby strengthening the spatial rationality of the skeleton structure. An affinity matrix is ​​constructed by using a skeleton connection matrix and a learnable topology matrix, enhancing both spatial rationality and temporal coherence. Furthermore, prior biases are superimposed in the attention calculation, balancing data-driven approaches and kinematic constraints, thus improving the model's robustness and accuracy. Particularly in sign language motion optimization tasks, this method significantly improves the smoothness of fine hand movements. Compared to traditional methods, the optimized 3D motion data is more accurate in detail processing, especially in the expression of rapid gestures or subtle movements, where accuracy is significantly improved. More importantly, although the optimized effect approaches the accuracy of complex models, the simplified model structure also significantly improves computational efficiency, achieving a balance between high accuracy and high efficiency. This is particularly important when real-time or large-scale processing of sign language motion data is required, providing higher computational efficiency while ensuring accuracy, meeting practical application needs. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the data optimization model framework that integrates spatiotemporal prior attention in an embodiment of the present invention;

[0029] Figure 2 This is a detailed diagram of the data optimization model that integrates spatiotemporal prior attention in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart illustrating the prior and coding fusion of motion data feature space in an embodiment of the present invention.

[0031] Figure 4 This is a block diagram of motion data feature temporal prior and encoding fusion in an embodiment of the present invention;

[0032] Figure 5 This is a block diagram of spatial prior and spatial multi-head attention fusion in an embodiment of the present invention;

[0033] Figure 6 This is a block diagram of temporal prior and temporal multi-head attention fusion in an embodiment of the present invention;

[0034] Figure 7 This is a block diagram of spatiotemporal attention fusion and regression head output in an embodiment of the present invention;

[0035] Figure 8 This is a smooth curve diagram of the palm in a specific embodiment of the present invention;

[0036] in Figure 8 (a) is a comparison of the smooth curves before and after optimization of the X-coordinate position of the palm; (b) is a comparison of the smooth curves before and after optimization of the Y-coordinate position of the palm; and (c) is a comparison of the smooth curves before and after optimization of the Z-coordinate position of the palm.

[0037] Figure 9 This is a smooth curve diagram of the wrist joint in a specific embodiment of the present invention;

[0038] in Figure 9 (a) is a comparison of the smooth curves before and after optimization of the X-coordinate position of the wrist joint; (b) is a comparison of the smooth curves before and after optimization of the Y-coordinate position of the wrist joint; and (c) is a comparison of the smooth curves before and after optimization of the Z-coordinate position of the wrist joint. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0040] Example 1

[0041] To address the shortcomings of current video-based 3D reconstruction technologies, such as insufficient accuracy in reconstructing complex movements due to ambient light interference and occlusion, and the tendency of existing methods to focus primarily on body parts while neglecting hand movements, this invention proposes a model framework that jointly trains temporal and spatial priors. This framework fuses temporal and spatial priors with the input data, and further integrates spatial and temporal prior attention modules in the multi-head attention mechanism. This method combines spatial and temporal priors with the input data for joint training, and further integrates spatial and temporal prior attention in the multi-head attention mechanism. This achieves improved accuracy and smoothness optimization of 3D motion data extracted from human motion videos, laying the foundation for higher-precision 3D reconstruction. Experiments on the CMU dataset demonstrate that this invention exhibits significant advantages in comparative experiments, with an average error reduction of 14.09 mm. Furthermore, it shows progress in optimizing the smoothness of fine movements such as finger joints, providing anatomically sound and temporally consistent 3D motion data support for the reconstruction of complex sign language movements.

[0042] like Figure 1 As shown, the dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention includes the following steps:

[0043] Step 1. Obtain the 3D motion data sequence. Use the 3D coordinates of each joint contained in each frame of the 3D motion data sequence as the 3D motion data to be optimized, and construct the training dataset.

[0044] For raw video or motion capture data, the first step is data cleaning to remove noise and invalid frames (e.g., invalid or incomplete data may appear at the beginning or end of an action). During training, ensure that the joint data at each time step is aligned to the same temporal sequence (i.e., the joint data length is the same at each time point). If the number of frames in some motion sequences is inconsistent, time step interpolation or time step pruning can be performed to ensure that each motion sequence has a uniform temporal length and to remove missing or incomplete joint data to avoid affecting the training effect.

[0045] In this embodiment of the invention, two datasets were used for training and testing, respectively.

[0046] I. The CMU dataset is a commonly used 3D human pose dataset containing rich motion capture data, covering various types of movements (such as running, jumping, raising hands, etc.). This dataset provides the 3D coordinates of various joints of the human body at different time steps, making it suitable for training and evaluating models based on 3D motion data.

[0047] During training, this invention uses the 3D position data of each frame and each joint from the CMU dataset as labels. These labels are directly used for model training, optimizing the model through supervised learning. After training, the model is used in the testing phase for validation on a sign language dataset collected by DeepMotion software.

[0048] II. DeepMotion software was used to extract 3D motion data from sign language videos. The sign language videos were from the Chinese Sign Language dataset. The software then performed video analysis to identify keyframes and joint position information in the sign language, and finally saved the 3D position of each joint as a BVH file. The BVH (Biovision Hierarchy) file format contains the skeleton hierarchy and joint position data for each frame, making it suitable for processing and analyzing 3D pose data.

[0049] Labeled data from the CMU dataset, containing the 3D position of each joint in each frame, was used to train the model, enabling it to learn how to optimize and predict joint 3D coordinates. This optimized model was then tested on a sign language dataset collected by DeepMotion software. During testing, the model optimized and predicted the joint 3D coordinates for each frame, outputting the optimized joint position. The performance and effectiveness of the optimized model on the new data were validated by comparing the optimization results on the training data (CMU dataset) and the test data (sign language dataset). Specifically, the model optimized the joint 3D coordinates in the sign language dataset by reducing noise, smoothing movements, and maintaining temporal consistency.

[0050] Step 2. Build a data optimization model that integrates spatiotemporal prior attention. The overall framework of the model is as follows: Figure 1 As shown, 3D motion data is trained using a joint method, incorporating spatial and temporal prior information. The model includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head.

[0051] This invention addresses the jitter and instability issues in 3D pose prediction data by proposing a model framework that integrates temporal and spatial priors with attention. In this framework, temporal and spatial priors are fused with the input data, and spatial prior attention and temporal prior attention are integrated into the multi-head attention component to optimize 3D motion data. This enhances the model's ability to model joint spatial structure and temporal dependencies, resulting in smoother 3D motion data that conforms to kinematic constraints.

[0052] First, the three-dimensional motion data to be optimized is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively.

[0053] High-dimensional input features containing kinematic and temporal prior knowledge are augmented with positional and temporal codes, respectively, to provide the model with additional positional information and temporal order. The high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are then linearly fused using spatiotemporal priors to integrate spatial and temporal information.

[0054] Next, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to perform spatial multi-head attention and temporal multi-head attention processing to capture more granular spatiotemporal dependencies and further enhance the interaction and dependence between spatial and temporal features.

[0055] After attention modeling in two stages—spatial and temporal multi-head attention—the output undergoes normalization and feedforward feedback (FFN) processing to further optimize the feature representation. Then, the output features of the spatial and temporal multi-head attention modules are linearly fused through spatiotemporal multi-head attention and input to the regression head for mapping, ultimately obtaining the 3D coordinate output of each joint in each time frame, i.e., the optimized 3D motion data.

[0056] As can be seen from the above process, in the method of this invention, the spatial prior and temporal prior modules process the motion data in a dual-path parallel manner. This parallel structure can enhance the model's learning ability in both spatial and temporal dimensions, and ultimately fuse the features of both to obtain more accurate motion predictions. The entire optimization process is divided into two main branches:

[0057] 1. Spatial Prior Processing Branch: a. The input 3D motion data first passes through a spatial prior module, which models the spatial dependencies between joints using a skeleton connectivity matrix and a learnable topological matrix. This allows the model to understand the relationships between joints in a spatial dimension, thereby enhancing its ability to capture these spatial relationships. b. Position encoding is added to the spatial features, introducing positional information to help the model maintain the spatiotemporal order of the joints. The spatial features are further processed by a spatial multi-head attention module to improve the interaction and dependencies between spatial features of the joints.

[0058] 2. Temporal Prior Processing Branch: a. On the other hand, the input data passes through the temporal prior module, which focuses on modeling the temporal changes of joints to capture temporal consistency. Through this step, the model can smooth actions, reduce temporal jitter, and ensure the consistency of temporal data. b. Temporal encoding introduces temporal information to help the model understand the order of each time step. The temporally processed features are further optimized through the temporal multi-head attention module to enhance the interaction between temporal features.

[0059] In this process, the spatial prior branch and the temporal prior branch work in parallel to optimize spatial and temporal features, respectively. The outputs of the two branches are then linearly fused and merged later to generate a comprehensive spatiotemporal feature representation, providing richer spatiotemporal information for subsequent multi-head attention modules. After linearly fusing the outputs of the spatial and temporal prior branches, the model continues with dual-path parallel optimization to further enhance the interactivity and dependency of spatial and temporal features. Specifically, on the fused features, multi-head attention calculations, normalization, and front feedback layer (FFN) processing are performed separately for both spatial and temporal aspects to further improve the model's spatiotemporal representation capabilities.

[0060] 1. Parallel processing of spatial and temporal multi-head attention. Independent optimization of spatial and temporal features: The fused features are first fed into separate spatial and temporal multi-head attention modules, which independently optimize the spatial and temporal features respectively. At this stage: the spatial multi-head attention module further enhances the interaction and dependency of spatial features, focusing on the spatial relationships between joints. The temporal multi-head attention module independently optimizes temporal features, ensuring dynamic consistency between temporal features. This independent optimization process ensures that spatial and temporal information is expressed and processed to the maximum extent in their respective dimensions.

[0061] 2. Normalization and Feed-Forward Network (FFN) Processing. Normalization: After multi-head attention computation, the feature output undergoes layer normalization (LN). Normalization helps stabilize model training, ensuring a smooth feature distribution at each time step and joint, thereby improving the model's convergence speed and training performance. Feed-Forward Network (FFN): Subsequently, the normalized spatial and temporal features are processed separately through independent feed-forward networks (FFNs). FFN layers are used to perform non-linear transformations on the features, increasing the model's expressive power and non-linear mapping capabilities. Each feature is further optimized after passing through the FFN, thereby enhancing the understanding and modeling of spatiotemporal features. In this stage, spatial and temporal features are independently optimized under the processing of multi-head attention, normalization, and FFN, ensuring that each feature can be individually learned and strengthened by deep learning.

[0062] Through this dual-path parallel optimization, the model can fully understand and integrate information from both spatial and temporal dimensions, resulting in a more accurate and stable final output. After dual-path parallel optimization involving spatial and temporal multi-head attention, normalization, and a pre-feedback layer, the model merges spatial and temporal features and finally generates the 3D coordinate output of each joint across all time frames via a regression head, in the shape of... , Represents the number of frames in time. This represents the number of joints, and 3 represents the three-dimensional coordinates of each joint.

[0063] This dual-path parallel optimization method enables the proposed model to learn the interaction and dependence of spatial and temporal features independently and efficiently, and improves the accuracy of prediction through the fused spatiotemporal features.

[0064] The following is in conjunction with the appendix Figure 2 The processing flow of the data optimization model that integrates spatiotemporal prior attention is explained in detail.

[0065] This invention uses a joint training method to simultaneously train the input three-dimensional motion data. Optimize the spatial prior attention and temporal prior attention modules, among which... .in yes The Middle indivual The matrix represents the first... 3D motion data of the frame , For frame number, The number of joints.

[0066] Figure 3 This is a framework for fusing prior knowledge and coding in the feature space of motion data. By integrating kinematic and trajectory prior knowledge into the framework structure, it overcomes the limitations of traditional methods in modeling joint constraints in 3D human pose estimation.

[0067] Simultaneously, human anatomical priors are introduced in a weighted manner during the transformation of joint features to enhance the model's ability to express kinematic constraints, thereby improving the modeling accuracy for complex human movements and postures. In the motion data feature space encoding part, the input three-dimensional motion data is first mapped to a high-dimensional feature space through a fully connected layer linear transformation.

[0068] Define the 3D motion data to be optimized for the i-th frame. Joint feature matrix after linear transformation The formula is as follows:

[0069] ;

[0070] In the formula for of Learnable weight matrix for Bias matrix, For feature dimensions.

[0071] The spatial prior module injects prior knowledge of human skeletal structure into this high-dimensional representation.

[0072] definition Skeletal connection matrix Skeletal connection matrix elements in for:

[0073] ;

[0074] To enable the model to also perceive global dependencies between non-connected joints, define Learnable topological matrix Topological matrices can be learned elements in These are trainable parameters, representing the joints. and The strength of the dependency between them.

[0075] Learnable topological matrices Connection matrix with skeleton Linear combination, the formula is as follows:

[0076] ;

[0077] In the formula for Spatial affinity matrix; It is a hyperparameter representing the weighted average between skeletal connectivity and skeletal motion. For the ... Frame high-dimensional input features Integrating spatial priors, defining matrix :

[0078] ;

[0079] In the formula, for Learnable projection matrix, It is a high-dimensional input feature that includes prior kinematic knowledge.

[0080] In this way, each joint can not only utilize the skeletal connections It can aggregate features from directly connected bones and also utilize the skeletal motion relation matrix. Obtaining information from all joints enhances the ability to model spatial relationships.

[0081] Further analysis of high-dimensional input features that incorporate prior kinematic knowledge The location encoding process is as follows:

[0082] definition Fixed position encoding matrix Its even-numbered sequences are sine functions, and its odd-numbered sequences are cosine functions, expressed as:

[0083] ;

[0084] ;

[0085] In the formula It is an index of the key point location. It is an index of the feature dimension. It is the feature dimension; It is the first in PE line, number Column data; It is the first in PE line, number The data in the column.

[0086] Fixed position encoding matrix and high-dimensional input features containing prior kinematic knowledge To add them together, the formula is:

[0087] ;

[0088] in This indicates the position-encoded result. High-dimensional space features.

[0089] Then, the high-dimensional space features after position encoding are processed. Stacking, to obtain Stacked matrix :

[0090] .

[0091] Figure 4 The diagram shows the fusion of temporal priors and encoding for motion data features. It integrates the temporal prior information of joint movements into 3D human pose data. The model mainly improves the motion coherence in the video by enhancing temporal consistency.

[0092] The core of this stage is to incorporate the dynamic characteristics of human movements as a temporal prior into the overall structure, enabling the model to better capture temporal dependencies and action continuity across frames. Specifically, by weighting the correlation of joint sequences in the temporal dimension, the network can establish key dynamic connections between different frames, thereby helping to suppress jitter and abrupt changes.

[0093] In the temporal encoding part of the motion data features, the first step is to process all the input 3D motion data to be optimized. Transpose the matrix to obtain the transposed matrix. : .

[0094] in This represents the 3D motion data of the i-th frame, with dimensions J×3. This represents all input 3D motion data, with dimensions C×J×3 (C represents the number of frames, J represents the number of joints, and 3 represents the dimension).

[0095] in yes The Middle indivual The matrix represents the first... Motion data of key points, .

[0096] Similarly, the transposed 3D motion data to be optimized is linearly transformed into a high-dimensional feature space through a fully connected layer.

[0097] Definition of the first Motion data at key points Joint feature matrix after linear transformation for:

[0098] ;

[0099] In the formula for of Learnable weight matrix for Bias matrix, For feature dimensions.

[0100] Then define a Adjacent time frame topology matrix Used to capture the adjacency relationship between consecutive frames.

[0101] The topology matrix of adjacent time frames elements in for:

[0102] .

[0103] like If it is 1, then the first Frame and the Frames are directly adjacent to each other; otherwise, they are not directly adjacent. The topology matrix of adjacent time frames helps to learn the dependencies between consecutive actions and enhances the model's learning ability in the temporal dimension.

[0104] To further enhance the model's ability to learn dependencies across time frames, a... Learnable time topology matrix This indicates the dependency between distant frame pairs, which is helpful for modeling long sequences and complex actions.

[0105] The model learns the correlation between any pair of frames through training.

[0106] Topology matrix of adjacent time frames and learnable time topology matrix The weighted fusion formula is as follows:

[0107] ;

[0108] In the formula Indicates the obtained Time affinity matrix; is a hyperparameter representing the weighted sum of the topology matrices of adjacent time frames and the learnable temporal topology matrix. Subsequently, the temporal prior module applies the weighted sum of the first... High-dimensional input features of key points By incorporating temporal priors, global temporal dependencies are integrated into feature computation, as shown in the following formula:

[0109] ;

[0110] In the formula, It is a high-dimensional input feature that incorporates prior temporal knowledge. for The learnable projection matrix allows for the perception of dependencies in more distant frames while preserving smooth prior knowledge of adjacent frames.

[0111] Further analysis of high-dimensional input features that incorporate temporal prior knowledge The time encoding process is as follows:

[0112] definition Fixed position encoding matrix Its even-numbered sequences are sine functions, and its odd-numbered sequences are cosine functions, expressed as:

[0113] ;

[0114] ;

[0115] In the formula, It is the index of the time frame. It is an index of the feature dimension. It is the feature dimension. yes The Middle line, number Column data; yes The Middle line, number The data in the column.

[0116] Fixed position encoding matrix and high-dimensional input features containing prior temporal knowledge Add them together to get the position-encoded result. High-dimensional features The formula is as follows:

[0117] ;

[0118] Then, the high-dimensional features after position encoding are processed. Stack and transpose to obtain Stacked matrix :

[0119] ;

[0120] Stacked matrices incorporating spatial priors and stacked matrices incorporating time priors After undergoing a linear transformation by a fully connected layer, the components are merged together, as expressed in the following formula: ;

[0121] In the formula, for The fusion matrix, and These are the trainable weight parameters.

[0122] Define a fusion matrix incorporating spatial and temporal priors. The fusion matrix, which incorporates spatial and temporal priors, is further processed through spatial multi-head attention modules and temporal multi-head attention modules, respectively.

[0123] in It is a fusion matrix The Middle indivual The matrix represents the first... Motion data of frames, .

[0124] like Figure 5 As shown, the processing flow of the spatial multi-head attention module is as follows:

[0125] I. Spatial attention head query, key, and value calculation;

[0126] The spatial multi-head attention module employs a spatial multi-head self-attention (MHSA) mechanism to process the joint feature matrix. The process involves processing each attention head through a linear transformation matrix, mapping joint features differently. This allows the model to capture relationships between joints from multiple perspectives, thereby capturing different attention patterns and enhancing spatial dependencies between features. Separate definitions are provided. matrix , , , used to represent the first The query, key, and value after the attention head is mapped ,in:

[0127] ; ; ;

[0128] In the formula , , They are The The first attention head to the first The learnable linear projection matrix of the frame.

[0129] II. Integration of spatial priors in multi-head attention layers;

[0130] During the computation of spatial attention heads, the multi-head attention layer passes through... During normalization and weighted output, spatial prior knowledge is incorporated, and the spatial affinity matrix is ​​used. It is integrated into the multi-head attention layer; based on the standard Transformer's calculation formula, it has been improved. The similarity score of the attention weights is not only obtained from the inner product of Query and Key, but also incorporates kinematic prior weights into the attention score of each Transformer layer, that is, a structural prior is superimposed. The Query-Key part ensures dynamic feature similarity modeling. This ensures the topological constraints of the skeleton. In this way, each attention layer can learn data-driven relevance while also adhering to prior knowledge of human kinematics, resulting in a weighted output:

[0131] .

[0132] In the formula, It is a size of The matrix, These are hyperparameters that control the spatial prior attention weights. The attention scores are normalized so that the weighted scores for each value are between [0,1] and the sum is 1.

[0133] if If a value of 0 at a certain position in the joint indicates that this joint has no function in information transmission, meaning there is no direct structural constraint between the two joints. If a certain position has a non-zero weight value (e.g., 1), it means that there is a kinematic connection between the two joints, which can adjust the intensity of attention and enhance the attention score. T is the transpose operation.

[0134] By concatenating the outputs of all spatial attention heads, a richer representation is obtained, as shown in the following formula:

[0135] .

[0136] In the formula For size The matrix represents the attention-weighted features, reflecting the information learned by the model through the attention mechanism between various joints. Let be a linear transformation matrix.

[0137] By stacking multiple layers of Spatial Multi-Head Self-Attention (MHSA), the modeling capability of the model can be further improved, and the model can obtain a higher level of global spatial correlation among all key points in the same frame.

[0138] III. Layer normalization and front-feedback layer processing;

[0139] Will Continuing with the subsequent processing of the Transformer, the first step is layer normalization, defining... matrix :

[0140] ;

[0141] In the formula, and For input The mean and variance, and These are learnable parameters.

[0142] Then through the front feedback layer and layer normalization The process yields a high-dimensional joint feature tensor that incorporates spatial priors:

[0143] ; ;

[0144] In the formula, To integrate spatial a priori High-dimensional motion data.

[0145] like Figure 6 As shown, the processing flow of the temporal multi-head attention module is as follows:

[0146] First, the fusion matrix First, perform a transpose operation to obtain the transposed matrix. The formula is expressed as follows:

[0147] ;

[0148] In the formula, yes The Middle indivual The matrix represents the first... Motion data of key points, .

[0149] I. Time-based attention to head queries, key-value calculations.

[0150] Further processing is performed using a temporal multi-head attention module, which employs a temporal multi-head self-attention (Temporal MHSA) mechanism to process the joint feature matrix. Further processing of temporal features enhances the model's ability to capture dynamic changes, allowing it to understand the relationships between frames from different perspectives. This helps the model capture the consistency and dynamic changes between frames. (Define separately) matrix , , , respectively representing the first The query, key, and value after mapping of each attention head:

[0151] ; ; ;

[0152] , , They are The The first attention head to the first Learnable linear projection matrices for each key point.

[0153] II. Incorporation of temporal priors in multi-head attention layers;

[0154] Similar to the spatial prior attention part above, temporal prior knowledge is integrated into the multi-head attention layer, incorporating the temporal affinity matrix. It is integrated into the multi-head attention layer. When calculating the temporal prior attention heads, the calculation formula of Transformer is also improved, incorporating kinematic prior weights into the attention score of each Transformer layer. In other words, when calculating the similarity score of the attention weights here, in addition to relying on the inner product of Query and Key to model the dynamic similarity between temporal features, a temporal prior weight is also introduced into the attention score of each layer of Transformer. This is equivalent to superimposing a temporal prior bias into the attention calculation. The Query-Key part ensures the feature dependencies between different time frames, while the temporal prior guarantees the continuity and dynamic smoothness between frames. In this way, each attention layer can capture the temporal correlation of the data while adhering to the temporal prior constraints of human motion. Its attention calculation is as follows:

[0155] ;

[0156] In the formula, It is a size of The matrix, This is a hyperparameter used to control the strength of the influence of temporal prior on attention weights. It is the dimension of the key vector; Its function is to scale the similarity, so that the attention weights can be stably normalized. Within the range, and ensure that the sum is 1.

[0157] If a certain time frame is If the weight in the time frame is 0, it means that the time frame does not contribute to the information transmission, that is, there is no direct temporal dependency between the frame and the current frame. If it is a non-zero weight value (such as 1), it indicates that the frame and the current frame have a significant temporal relationship, thereby enhancing its role in attention calculation. T is the transpose operation.

[0158] By concatenating the outputs of all time-attention heads, a richer representation can be obtained:

[0159] ;

[0160] In the formula, For size The matrix represents the attention-weighted features, reflecting the information learned by the model through the attention mechanism across different time frames. Let be a linear transformation matrix.

[0161] Similarly, by stacking multiple layers of Temporal Multi-Head Self-Attention (MHSA), the model can capture the dynamic dependencies between frames at different scales, thereby learning smooth and coherent motion trajectories. Through this module, the model can also better understand the temporal evolution characteristics of human movements.

[0162] III. Layer normalization and front-feedback layer processing.

[0163] Will Continuing with the subsequent processing of the Transformer, the first step is layer normalization, defining... matrix :

[0164] ;

[0165] In the formula, and For input The mean and variance, and These are learnable parameters;

[0166] Then through the front feedback layer and layer normalization The process yields a high-dimensional joint feature tensor with fused temporal priors:

[0167] ; ;

[0168] In the formula, For the fusion of time priors Transpose matrix of high-dimensional motion data.

[0169] like Figure 7 As shown, the process of spatiotemporal multi-head attention linear fusion and regression head output is as follows:

[0170] The output after passing through the spatial multi-head attention module Output of the temporal prior attention module Linear fusion is performed through fully connected layers to obtain a high-dimensional feature tensor that includes both human skeleton topological constraints and cross-frame dependencies and temporal consistency. This is defined as follows: fusion matrix The formula is as follows:

[0171] ;

[0172] The fused output is then further input into the regression head for processing, as shown in the following formula:

[0173] ;

[0174] in, yes The Middle indivual The matrix, ;

[0175] In the regression head, a fully connected layer performs a linear transformation to map the high-dimensional feature space to three dimensions, resulting in a smooth three-dimensional coordinate output from the model. Matrix after linear transformation The formula is as follows:

[0176] ;

[0177] In the formula for Learnable weight matrix for Bias matrix;

[0178] Finally, the linearly transformed data is concatenated to obtain the final optimized and smoothed 3D motion data output, which is the 3D coordinate output of each joint in each time frame, as shown in the following formula:

[0179] ;in for The matrix.

[0180] Step 3. Train the model using the training dataset from Step 1, and then optimize the model for motion data. The model training process is fairly standard. After training is complete, the model is deployed.

[0181] The input data is a sequence of 3D motion data to be optimized. ,in For time frames, This represents the number of joints, where 3 represents... Coordinates. The processing flow is as follows:

[0182] First, the input data to be optimized passes through a spatial prior module, which models spatial dependencies using a skeleton connection matrix and a learnable topological matrix, thereby enhancing the model's understanding of spatial relationships between joints.

[0183] Meanwhile, the input data is fed into the time prior module to model the changes and temporal consistency of the joints over time. This time prior module can help the model smooth the movements and reduce temporal jitter.

[0184] Next, positional encoding and temporal encoding incorporate positional and temporal information into the features, helping the model maintain spatiotemporal order and ensuring that each frame of the joint is processed in the correct spatiotemporal order.

[0185] Then, the outputs of the spatial and temporal prior modules are linearly fused and merged into a unified feature representation, providing a comprehensive spatiotemporal feature for the subsequent attention mechanism.

[0186] The fused features are then fed into the spatial multi-head attention module and the temporal multi-head attention module (which fuses spatial and temporal prior information respectively), further enhancing the interaction and dependence between spatial and temporal features.

[0187] Finally, the features processed by multiple attention steps are mapped through a regression head (usually a fully connected layer) to obtain optimized, smooth 3D motion data output, i.e., the 3D coordinate output of each joint in each time frame, with the shape of... .

[0188] In addition, to verify the effectiveness of the method of the present invention, the following experimental process and result analysis are also given.

[0189] This invention evaluated our model on the CMU public dataset. The CMU dataset, built by Carnegie Mellon University, is a widely used dataset in computer vision, graphics, and robotics, containing over 2600 different motion sequences covering actions such as walking, running, jumping, and dancing. The evaluation metrics used were PA-MPJPE and Accel.

[0190] PA-MPJPE (Procrustes-Aligned Mean Per-Joint Position Error) measures attitude structure error and evaluates the average distance between the predicted joint position and the actual joint. Unlike MPJPE, PA-MPJPE performs a rigid alignment with a rigid body and scale before calculating the average distance error to remove global scale, rotation, and translation differences. This better reflects the accuracy of the predicted attitude structure. The unit is consistent with the input coordinate unit (commonly mm).

[0191] Accel (Mean Per-Joint Acceleration Error) is used to measure the smoothness jitter of time. It evaluates the smoothness of time series by comparing the discrete second derivative (i.e., acceleration) of the predicted and ground truth trajectories. The smaller the value, the smoother the trajectory and the less jitter. The unit is coordinate units / (time unit²). If the frame rate is converted to seconds, it is commonly found in mm / s².

[0192] like Figure 8and Figure 9 As shown, where Figure 8 In the figure, (a), (b), and (c) represent the optimized coordinate trajectories of the hand's X, Y, and Z positions, respectively. Figure 9 In the diagram, (a), (b), and (c) represent the coordinate trajectories of the wrist joint before and after optimization at the X, Y, and Z positions, respectively. The red curve represents the original coordinate trajectory of the 3D motion data, while the green curve represents the optimized trajectory result. Comparative analysis shows that the overall motion sequence is significantly improved in terms of smoothness after optimization, effectively reducing jitter and discontinuities in the original data. The improvement in smoothness is particularly noticeable in hand movements, showing a better optimization effect compared to the wrist joint. This indicates that the method of this invention can better capture the motion characteristics of detailed parts such as the hand while ensuring the consistency of the overall motion, thereby improving the quality and naturalness of the 3D motion data.

[0193] In addition, this invention also compares and analyzes the effects of two other optimization algorithms. One is the One-EuroFilter method, which is essentially an optimization method based on first-order low-pass filtering. It suppresses the influence of noise signals during human motion tracking and achieves a balance between jitter and lag through adjustable parameters. The other method is SmoothNet, which uses a dedicated temporal refinement network to handle jitter caused by factors such as occlusion, thereby improving the continuity and stability of the frame sequence in the temporal dimension. The comparison results of the three methods are shown in Table 1.

[0194] Table 1 Comparison of results from different methods

[0195]

[0196] As shown in Table 1, the proposed method shows a significant improvement over the One-Euro Filter in terms of PA-MPJPE metrics, reducing the average error between the predicted and actual positions by 14.09 mm. It also reduces the Accel metric by 0.52 mm / s², indicating that the proposed method has advantages in noise suppression and improving temporal smoothness. While the proposed method is slightly less effective in reducing error compared to SmoothNet, the overall performance is very close. Furthermore, due to its simpler model structure, the proposed method is more efficient in terms of computational complexity compared to SmoothNet. This demonstrates that the proposed optimization method can effectively improve the smoothness and stability of 3D motion data while balancing performance and complexity.

[0197] This invention proposes a method for optimizing 3D sign language motion data by fusing temporal prior attention and spatial prior attention Transformer models, applying two novel prior attention mechanisms to 3D pose estimation.

[0198] Experimental results demonstrate that the method of this invention can suppress noise and effectively improve the temporal smoothness and stability of 3D motion data. This invention exhibits superior accuracy and error control in 3D motion data optimization, reducing the average PA-MPJPE error by 14.09 mm and the Accel index by 0.52 mm / s², effectively suppressing noise and achieving more accurate positioning. The invention improves temporal smoothness and motion coherence through a "local + global" temporal topology matrix, particularly effective for fine movements such as sign language. It integrates rigid and learnable spatial matrices, balancing anatomical rationality and global dependencies to avoid posture deviations. Specific optimizations are made for the high degree of freedom and strong occlusion of the hand, filling gaps in hand motion reconstruction. The model has a simple structure, high computational efficiency, and performance approaching that of complex models, making it suitable for real-time applications. Furthermore, by incorporating spatial and temporal priors, it enhances robustness to complex scenes such as occlusion and lighting changes, resulting in more stable and reliable data optimization results.

[0199] This invention proposes a dual-priority attention module that organically integrates spatial and temporal prior constraints to enhance the Transformer's ability to model joint structures and temporal dependencies. Specifically, spatial prior constraints construct an affinity matrix using the skeleton connectivity matrix and a learnable topology matrix, strengthening the spatial rationality of the skeletal structure; temporal prior constraints enhance the coherence and consistency of temporal data through modeling time series. This fusion not only effectively captures spatial dependencies between human joints but also handles dynamic changes in the temporal dimension, giving the model stronger expressive power in both spatiotemporal dimensions. Furthermore, superimposing prior biases in attention computation balances data-driven learning and kinematic constraints during optimization. This combination allows the model to not only rely on joint motion patterns in the training data but also effectively guide the model to follow the inherent structure and temporal continuity of the human skeleton, thereby improving the model's robustness and accuracy. The introduction of prior biases allows the model to rationally adjust the relationships between joints from a physical and anatomical perspective, reducing noise interference during training and improving the overall motion data optimization effect. Especially in sign language motion optimization tasks, this method significantly improves the smoothness of fine hand movements. Compared to traditional methods, the optimized 3D hand data demonstrates greater precision in detail processing, particularly in the representation of rapid gestures or subtle movements, where accuracy is significantly improved. More importantly, while the optimized results approach the accuracy of complex models, the simplified model structure also significantly enhances computational efficiency, achieving a balance between high precision and high efficiency. This is especially crucial when processing sign language motion data in real-time or on a large scale, providing higher computational efficiency while maintaining accuracy to meet practical application needs. In summary, this invention not only enhances the model's ability to depict spatial and temporal features and optimizes the smoothness of hand and overall movements, but also strikes an ideal balance between complexity and efficiency, providing an efficient and accurate solution for motion data optimization, especially sign language motion optimization.

[0200] Example 2

[0201] This embodiment 2 describes a dual-path parallel motion data optimization system that integrates spatiotemporal priors and attention. This system is based on the same inventive concept as the aforementioned dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention.

[0202] The dual-path parallel motion data optimization system that integrates spatiotemporal priors and attention in this embodiment includes the following modules:

[0203] The preprocessing module is used to acquire the three-dimensional motion data sequence, take the three-dimensional coordinates of each joint contained in each frame of the three-dimensional motion data sequence as the three-dimensional motion data to be optimized, and construct the training dataset.

[0204] And a data optimization module, used to build a data optimization model that integrates spatiotemporal prior attention, which includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head;

[0205] First, the three-dimensional motion data to be optimized is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively.

[0206] The high-dimensional input features containing kinematic and temporal prior knowledge are added with positional and temporal codes respectively, and the high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are linearly fused with spatiotemporal priors.

[0207] Subsequently, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to further enhance the interaction and dependence between spatial and temporal features;

[0208] Then, the output features of the spatial multi-head attention module and the temporal multi-head attention module are linearly fused through spatiotemporal multi-head attention and input into the regression head for mapping, finally obtaining the optimized and smooth 3D motion data output, that is, the 3D coordinate output of each joint in each time frame;

[0209] The model is trained using the training dataset, and the trained model is then used to optimize motion data.

[0210] It should be noted that the implementation process of the functions and roles of each functional module in the dual-path parallel motion data optimization system in this embodiment 2 is detailed in the implementation process of the corresponding steps in the method of embodiment 1 above, and will not be repeated here.

[0211] Example 3

[0212] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the dual-path parallel motion data optimization method fusing spatiotemporal priors and attention as described in embodiment 1 above.

[0213] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0214] Example 4

[0215] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, is used to implement the steps of the dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention in embodiment 1 above.

[0216] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0217] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A dual-path parallel motion data optimization method integrating spatiotemporal priors and attention, characterized in that, Includes the following steps: Step 1. Obtain the 3D motion data sequence, and use the 3D coordinates of each joint contained in each frame of 3D motion data in the 3D motion data sequence as the 3D motion data to be optimized, and construct the training dataset; Step 2. Build a data optimization model that integrates spatiotemporal prior attention, which includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head; First, the three-dimensional motion data to be optimized is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively. The high-dimensional input features containing kinematic and temporal prior knowledge are added with positional and temporal codes respectively, and the high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are linearly fused with spatiotemporal priors. Subsequently, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to further enhance the interaction and dependence between spatial and temporal features; Then, the output features of the spatial multi-head attention module and the temporal multi-head attention module are linearly fused through spatiotemporal multi-head attention and input into the regression head for mapping, ultimately obtaining optimized and smooth 3D motion data output; In step 2, the spatiotemporal multi-head attention linear fusion and regression head output process is as follows: The output after passing through the spatial multi-head attention module Output of the temporal prior attention module Linear fusion is performed through fully connected layers to obtain a high-dimensional feature tensor that includes both human skeleton topological constraints and cross-frame dependencies and temporal consistency. This is defined as follows: fusion matrix The formula is as follows: ； in Indicates the number of frames. The number of joints. For feature dimensions; The fused output is then further input into the regression head for processing, as shown in the following formula: ； in, yes The Middle indivual The matrix, ; In the regression head, a fully connected layer performs a linear transformation to map the high-dimensional feature space to three dimensions, resulting in a smooth three-dimensional coordinate output from the model. Matrix after linear transformation The formula is as follows: ； In the formula for Learnable weight matrix for Bias matrix; Finally, the linearly transformed data is concatenated to obtain the optimized, smooth 3D motion data output, which is the 3D coordinate output of each joint in each time frame, as shown in the following formula: ;in for Matrix; Step 3. Train the model using the training dataset, and then use the trained model to optimize motion data.

2. The dual-path parallel motion data optimization method integrating spatiotemporal priors and attention as described in claim 1, characterized in that, In step 2, the processing flow of the spatial prior module is as follows: First, the input 3D motion data to be optimized is linearly transformed into a high-dimensional feature space through a fully connected layer; the 3D motion data to be optimized in the i-th frame is defined. Joint feature matrix after linear transformation The formula is as follows: ； In the formula for of Learnable weight matrix for Bias matrix, For feature dimension, The number of joints; defined Skeletal connection matrix Skeletal connection matrix elements in for: ； definition Learnable topological matrix Topological matrices can be learned elements in These are trainable parameters, representing the joints. and The strength of the dependency between them; the learnable topological matrix Connection matrix with skeleton Linear combination; definition Spatial affinity matrix The formula is as follows: ； In the formula It is a hyperparameter that represents the weighted average between skeletal connectivity and skeletal motion. For the first Frame high-dimensional input features By integrating spatial priors, we obtain matrix The formula is: ； In the formula, for Learnable projection matrix, It is a high-dimensional input feature that includes prior kinematic knowledge.

3. The dual-path parallel motion data optimization method integrating spatiotemporal priors and attention as described in claim 1, characterized in that, In step 2, the processing flow of the time prior module is as follows: First, all the input 3D motion data to be optimized are processed. Transpose; Define the matrix obtained after transpose as , , yes The Middle indivual The matrix represents the first... Motion data of key points, ; Indicates the number of frames. The number of joints; Similarly, the transposed 3D motion data to be optimized is linearly transformed into a high-dimensional feature space through a fully connected layer; the first... Motion data at key points Joint feature matrix after linear transformation for: ； In the formula for of Learnable weight matrix for Bias matrix, For feature dimensions; then define a Adjacent time frame topology matrix Used to capture the adjacency relationship between consecutive frames; elements for: ； like If it is 1, then the first Frame and the Frames are directly adjacent to each other; otherwise, they are not directly adjacent to each other. Introduce a Learnable time topology matrix Topology matrix of adjacent time frames and learnable time topology matrix Weighted fusion yields Time affinity matrix The formula is as follows: ； In the formula is a hyperparameter representing the weighted sum of the topology matrices of adjacent time frames and the learnable temporal topology matrix; subsequently, the weighted sum of the topology matrices of the th time frame is applied. High-dimensional input features of key points By incorporating temporal priors, global temporal dependencies are integrated into feature computation, as shown in the following formula: ； In the formula, It is a high-dimensional input feature that incorporates prior temporal knowledge. for The learnable projection matrix.

4. The dual-path parallel motion data optimization method integrating spatiotemporal priors and attention as described in claim 1, characterized in that, In step 2, the high-dimensional input features containing prior kinematic knowledge are... The location encoding process is as follows: definition Fixed position encoding matrix Its even-numbered sequences are sine functions, and its odd-numbered sequences are cosine functions, expressed as: ； ； In the formula It is an index of the key point location. It is an index of the feature dimension. It is the feature dimension; It is the first in PE line, number Column data; It is the first in PE line, number Column data; Fixed position encoding matrix and high-dimensional input features containing prior kinematic knowledge To add them together, the formula is: ； in This indicates the position-encoded result. High-dimensional space features; Then, the high-dimensional space features after position encoding are processed. Stacking, to obtain Stacked matrix : ； in Indicates the number of frames; High-dimensional input features that include temporal prior knowledge The time encoding process is as follows: definition Fixed position encoding matrix Its even-numbered sequences are sine functions, and its odd-numbered sequences are cosine functions, expressed as: ； ； In the formula, It is the index of the time frame. It is an index of the feature dimension. It is the feature dimension. yes The Middle line, number Column data; yes The Middle line, number Column data; Fixed position encoding matrix and high-dimensional input features containing prior temporal knowledge Add them together to get the position-encoded result. High-dimensional features The formula is as follows: ； Then, the high-dimensional features after position encoding are processed. Stack and transpose to obtain Stacked matrix : ； Stacked matrices incorporating spatial priors and stacked matrices incorporating time priors After undergoing a linear transformation by a fully connected layer, the components are merged together, as expressed in the following formula: ； In the formula, for The fusion matrix, and These are the trainable weight parameters.

5. The dual-path parallel motion data optimization method integrating spatiotemporal priors and attention as described in claim 1, characterized in that, The processing flow of the spatial multi-head attention module is as follows: Define a fusion matrix incorporating spatial and temporal priors. ; in It is a fusion matrix The Middle indivual The matrix represents the first... Motion data of frames, ; I. Spatial attention head query, key, and value calculation; The spatial multi-head attention module employs a spatial multi-head self-attention mechanism to apply attention to the joint feature matrix. Perform processing; define matrix , , , used to represent the first The query, key, and value after mapping of each attention head: ； ； ； In the formula , , They are The The learnable linear projection matrix of an attention head for the i-th frame; II. Integration of spatial priors in multi-head attention layers; During the computation of spatial attention heads, the multi-head attention layer passes through... During normalization and weighted output, spatial prior knowledge is incorporated, and the spatial affinity matrix is ​​used. Integrate into the multi-head attention layer; The weighted output is: ； In the formula It is a size of The matrix, These are hyperparameters that control the spatial prior attention weights. The attention scores are normalized so that the weighted score of each value is between [0,1] and the sum is 1; By concatenating the outputs of all spatial attention heads, a richer representation is obtained, as shown in the following formula: ； In the formula For size The matrix represents the attention-weighted features, reflecting the information learned by the model through the attention mechanism between various joints. It is a linear transformation matrix; III. Layer normalization and front-feedback layer processing; Will Continuing with the subsequent processing of the Transformer, the first step is layer normalization, defining... matrix : ； In the formula, and For input The mean and variance, and These are learnable parameters; Then through the front feedback layer and layer normalization The process yields a high-dimensional joint feature tensor that incorporates spatial priors: ； ； In the formula, To integrate spatial a priori High-dimensional motion data.

6. The dual-path parallel motion data optimization method fusing spatiotemporal priors and attention as described in claim 1, characterized in that, The processing flow of the time-based multi-head attention module is as follows: Define a fusion matrix incorporating spatial and temporal priors. ; in It is a fusion matrix The Middle indivual The matrix represents the first... Motion data of frames, ; First, the fusion matrix First, perform a transpose operation to obtain the transposed matrix. The formula is expressed as follows: ； In the formula, yes The Middle indivual The matrix represents the first... Motion data of key points, ; I. Time-based attention head lookup, key, and value calculation; The temporal multi-head attention module employs a temporal multi-head self-attention mechanism to process the joint feature matrix. Processing timing characteristics; defining matrix , , , respectively representing the first The query, key, and value after mapping of each attention head: ； ； ； , , They are The The first attention head to the first Learnable linear projection matrices for each key point; II. Incorporation of temporal priors in multi-head attention layers; Integrating temporal prior knowledge into the multi-head attention layer, the temporal affinity matrix is ​​used. It has been integrated into the multi-head attention layer; Attention is calculated as follows: ； In the formula, It is a size of The matrix, This is a hyperparameter used to control the strength of the influence of temporal prior on attention weights. It is the dimension of the key vector; Its function is to scale the similarity, so that the attention weights can be stably normalized. Within the range, and ensure that the sum is 1; By concatenating the outputs of all time-attention heads, a richer representation can be obtained: ； In the formula, For size The matrix represents the attention-weighted features, reflecting the information learned by the model through the attention mechanism across different time frames. It is a linear transformation matrix; III. Layer normalization and front-feedback layer processing; Will Continuing with the subsequent processing of the Transformer, the first step is layer normalization, defining... matrix : ； In the formula, and For input The mean and variance, and These are learnable parameters; Then through the front feedback layer and layer normalization The process yields a high-dimensional joint feature tensor with fused temporal priors: ； ； In the formula, For the fusion of time priors Transpose matrix of high-dimensional motion data.

7. A dual-path parallel motion data optimization system for implementing the dual-path parallel motion data optimization method for fusing spatiotemporal priors and attention as described in claim 1, characterized in that, The dual-path parallel motion data optimization system that integrates spatiotemporal priors and attention includes the following modules: The preprocessing module is used to acquire the three-dimensional motion data sequence, take the three-dimensional coordinates of each joint contained in each frame of the three-dimensional motion data sequence as the three-dimensional motion data to be optimized, and construct the training dataset. And a data optimization module, used to build a data optimization model that integrates spatiotemporal prior attention, which includes a spatial prior module, a temporal prior module, a spatial multi-head attention module, a temporal multi-head attention module, and a regression head; First, the three-dimensional motion data is input into the spatial prior module and the temporal prior module respectively, which model the spatial structure and temporal relationship in the input data and output high-dimensional input features containing kinematic and temporal prior knowledge respectively. The high-dimensional input features containing kinematic and temporal prior knowledge are added with positional and temporal codes respectively, and the high-dimensional spatial features with added positional codes and the high-dimensional temporal features with added temporal codes are linearly fused with spatiotemporal priors. Subsequently, the fused features are fed into the spatial multi-head attention module that fuses spatial prior attention and the temporal multi-head attention module that fuses temporal prior attention, respectively, to further enhance the interaction and dependence between spatial and temporal features; Then, the output features of the spatial multi-head attention module and the temporal multi-head attention module are linearly fused through spatiotemporal multi-head attention and input into the regression head for mapping, ultimately obtaining optimized and smooth 3D motion data output; The model is trained using the training dataset, and the trained model is then used to optimize motion data.

8. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements the steps of the dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the steps of the dual-path parallel motion data optimization method that integrates spatiotemporal priors and attention as described in any one of claims 1 to 6.

Images