Human body action prediction method and system based on space-time interaction attention mechanism

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a human body action prediction method and system based on a space-time interaction attention mechanism. The method comprises the steps that a human body historical motion data set is obtained and preprocessed to obtain a historical motion sequence, the historical motion sequence is input into a Transform model based on a space-time interactive attention mechanism, firstly, time features and space features are extracted through a feature extraction branch, then the extracted features are interacted through the interactive attention mechanism, and the human body historical motion data set is obtained. And finally, obtaining a human motion sequence prediction result through the time features and the space features after joint interaction. According to the method, consistency and interaction information of 3D human body motion space and time characteristics are learned through an interaction attention mechanism, and more real 3D human body motion can be generated in the aspect of long-time modeling.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method and system for predicting human motion based on a spatiotemporal interactive attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Human motion prediction refers to the process of inferring future motion trajectories or postures based on existing motion data and specific conditions. This process involves using technologies such as computer vision, deep learning, and physical simulation to extract features from past motion patterns and predict the likelihood of a person's next step or a series of continuous actions. Human motion prediction is widely used in many fields. In sports training, it can help coaches and athletes analyze the rationality of movements and optimize training plans. In human-computer interaction, by predicting the user's movements, the device can respond in advance to improve the naturalness and fluency of the interaction. In the field of autonomous driving and robotics, by predicting the movement trajectory of pedestrians or other objects, the system can plan paths in advance to avoid collisions. In virtual reality (VR) and augmented reality (AR), human motion prediction makes the movements of virtual characters more realistic and improves the user's sense of immersion. In addition, in the field of medical rehabilitation, by predicting the patient's movements, it can help develop personalized rehabilitation plans and assist rehabilitation training.

[0004] Deep learning methods have significant advantages in the task of human motion prediction, improving the accuracy, flexibility and application scope of prediction. First, deep learning can automatically extract high-dimensional features from large amounts of data without relying on hand-crafted feature extraction methods. This allows the model to more accurately capture complex motion patterns and temporal relationships, especially when dealing with large-scale, high-dimensional motion data. Second, deep learning models, especially recurrent neural networks (RNNs), long short-term memory networks (LSTMs) and Transformers, are good at processing temporal data and can effectively model the dynamic characteristics of human motion. This enables them to predict continuous motion over long time spans and generate more natural action sequences. In addition, by using deep generative models such as generative adversarial networks (GANs), diverse motion predictions can be generated, taking into account different possibilities, thereby improving the robustness and generalization of the model. Deep learning methods also have good scalability and can easily integrate multiple input data types (such as vision, speech, sensor data), thereby improving the accuracy and applicability of predictions. In practical applications, such as virtual reality, intelligent monitoring, autonomous driving and robot motion planning, the powerful capabilities of deep learning provide a more intelligent and efficient solution for human motion prediction.

[0005] The Transformer mechanism was proposed by the Google team and used for machine translation. It has now become the main method for natural language processing tasks. The key part of the Transformer is the multi-head self-attention mechanism, which helps capture long-term dependencies while improving global modeling capabilities. Recent studies have shown that the Transformer network structure significantly improves the performance of various tasks, such as image classification, object detection, and visual question answering systems.

[0006] The Transformer model performs well in extracting spatiotemporal features. Through the self-attention mechanism, the Transformer can simultaneously focus on the relationship between different positions in the sequence, thereby capturing dependencies in the time dimension. At the same time, the Transformer can also use its multi-head attention mechanism to analyze the input data from different angles to obtain richer spatial features. When processing spatiotemporal data, the Transformer's time branch is responsible for capturing the dynamic changes of actions or events between different time frames, while the spatial branch focuses on the relationship between objects or human joints at different spatial positions. This dual-branch structure allows the Transformer to effectively integrate temporal and spatial information, thereby providing a more accurate and comprehensive feature representation for the task, and is widely used in fields such as human motion prediction and video analysis.

[0007] Although Transformer-based methods have achieved remarkable results, when dealing with the interaction of time and space information, these methods do not perform any operations to enhance the coherence between the two branches except for splicing the time and space information at the end of the model; existing methods all ignore the consistency of time and space information, which is an important clue when dealing with time-varying data. Summary of the invention

[0008] In order to solve the deficiencies of the prior art, the present invention provides a human motion prediction method, system, electronic device, computer-readable storage medium and computer program product based on a spatiotemporal interactive attention mechanism. To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a method for predicting human motion based on a spatiotemporal interactive attention mechanism, comprising: Obtain human historical motion data set and preprocess it to obtain historical motion sequence; After the acquired historical motion sequence is position-encoded, it is input into the Transformer model based on the spatiotemporal interactive attention mechanism. First, the temporal feature extraction branch and the spatial feature extraction branch are used to extract temporal features and spatial features respectively. Then, the extracted features are interacted using the interactive attention mechanism. Finally, the human motion sequence prediction result is obtained by combining the interactive temporal features and spatial features. The interactive attention mechanism is used to interact with the extracted features, including: using the self-attention mechanism to exchange information between a feature extraction branch and the patch tokens of another feature extraction branch; after obtaining information from the other feature extraction branch, the updated patch tokens are brought back to the own branch through the next spatiotemporal Transformer module, so as to pass the information learned from the other feature extraction branch back to the own branch, thereby enhancing the spatiotemporal representation capability.

[0009] Furthermore, the processing process of the Transformer model based on the spatiotemporal interactive attention mechanism includes: using N layers based on the Transformer attention mechanism to extract spatiotemporal features, and distinguishing the spatial features and the temporal features into two parts of feature information marks respectively. and feature blocks , mark the feature information in the space branch Characteristic information flags in time branches After dimensional alignment, the query vector Q is obtained by projection, and the spatial feature information mark and time feature blocks A cross conversion is performed between them to obtain the key vector K and the value vector V; after the query vector Q is multiplied by the key vector K and the value vector V, the feature information mark in the spatial branch after the dimension alignment is obtained. After completing the time interaction features, the spliced ​​features are back-projected and then combined with the spatial feature blocks. Splicing is performed to obtain the characteristics after information return; the consistent spatial characteristics after time-space interaction.

[0010] Furthermore, the temporal feature extraction branch is used to perform temporal feature extraction, including: utilizing the self-attention mechanism in Transformer to update its feature representation by considering the correlation between the same joint point in different time frames; and introducing a mask matrix in the calculation process of the self-attention mechanism to prevent future information leakage.

[0011] Further, the spatial feature extraction branch is used to extract spatial features, including: given a human posture embedding and weight matrix, using a multi-head attention mechanism to aggregate at time All joint information under the , get the spatial features.

[0012] Furthermore, the method of obtaining a human motion sequence prediction result by combining the time features and the space features after the interaction includes: fusing the time features and the space features after the interaction by adding them together, and then obtaining the prediction result by normalization.

[0013] Furthermore, the Adam optimizer is used to train the Transformer model based on the spatiotemporal interactive attention mechanism, and the parameters of the model are updated through back propagation.

[0014] In a second aspect, the present invention provides a human motion prediction system based on a spatiotemporal interactive attention mechanism, comprising: The data acquisition and preprocessing module is configured to: acquire a human body historical motion data set and perform preprocessing to obtain a historical motion sequence; The joint prediction module is configured to: after position encoding the acquired historical motion sequence, input it into the Transformer model based on the spatiotemporal interactive attention mechanism, firstly use the temporal feature extraction branch and the spatial feature extraction branch to extract temporal features and spatial features respectively, then use the interactive attention mechanism to interact with the extracted features, and finally obtain the human motion sequence prediction result by combining the interactive temporal features and spatial features; The interactive attention mechanism is used to interact with the extracted features, including: using the self-attention mechanism to exchange information between a feature extraction branch and the patch tokens of another feature extraction branch; after obtaining information from the other feature extraction branch, the updated patch tokens are brought back to the own branch through the next spatiotemporal Transformer module, so as to pass the information learned from the other feature extraction branch back to the own branch, thereby enhancing the spatiotemporal representation capability.

[0015] In a third aspect, the present invention provides an electronic device; An electronic device includes a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned human motion prediction method based on the spatiotemporal interactive attention mechanism.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium; A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the above-mentioned human motion prediction method based on a spatiotemporal interactive attention mechanism.

[0017] In a fifth aspect, the present invention provides a computer program product; A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned human motion prediction method based on a spatiotemporal interactive attention mechanism.

[0018] One or more of the above technical solutions have the following beneficial effects: (1) This paper proposes an interactive method based on the Transformer spatiotemporal attention mechanism to effectively predict 3D human motion. This method divides the human motion sequence information into a temporal branch and a spatial branch, and uses a Transformer attention method to extract features from the spatial and temporal branches of the human motion respectively, aiming to capture the global information of the human motion.

[0019] (2) The present invention designs a method based on the Transformer-based spatiotemporal interaction attention mechanism to learn the consistency and interactive information of the spatial and temporal features of 3D human motion. The spatiotemporal interaction attention mechanism method is divided into three stages, from the Transformer-based spatiotemporal information aggregation to the spatiotemporal information interaction, and finally to the spatiotemporal information returning to the respective spatiotemporal branches. Through this attention mechanism, the method designed by the present invention can produce more realistic 3D human motion in terms of long-term modeling.

[0020] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0022] Figure 1 This is a flow chart of a method for predicting human motion based on a spatiotemporal interactive attention mechanism according to a first embodiment.

[0023] Figure 2 This is a structural diagram of the Transformer model based on the spatiotemporal interactive attention mechanism of the first embodiment.

[0024] Figure 3 This is a flow chart of the interactive attention mechanism of the first embodiment; Figure 4 This is a diagram of the prediction results of the first embodiment. DETAILED DESCRIPTION

[0025] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0028] Embodiment 1 like Figure 1 , Figure 2 As shown, this embodiment discloses a human motion prediction method based on a spatiotemporal interactive attention mechanism, including: Step 1: Obtain the human body historical motion data set and perform data preprocessing to obtain the historical motion sequence; Step 2: After the acquired historical motion sequence is position-encoded, it is input into the Transformer model based on the spatiotemporal interactive attention mechanism. First, the temporal feature extraction branch and the spatial feature extraction branch are used to extract the temporal features and spatial features respectively. Then, the interactive attention mechanism is used to interact with the extracted temporal features and spatial features. Finally, the interactive temporal features and spatial features are jointly predicted to obtain the human motion sequence.

[0029] The step 1 includes: step 101: obtaining a human body historical motion data set; The present invention collects a public human motion dataset consisting of 3.6 million human postures and corresponding images captured by a high-speed motion capture system. The high-speed motion capture system uses four high-resolution progressive scanning cameras to obtain 50 Hz video data. The dataset contains the activities of 11 subjects in 17 scenarios such as discussion, smoking, taking pictures, and making phone calls, and provides accurate 3D joint positions and high-resolution videos.

[0030] The data set in the present invention is divided into a training set and a validation set. As shown in Table 1, the present invention uses subjects No. 1, No. 6, No. 7, No. 8, and No. 9 for training, and subjects No. 5 and No. 11 for testing.

[0031] Table 1 Number of training and validation samples in the human motion dataset

[0032] Step 102: data preprocessing; The purpose of skeleton-based human motion prediction is to predict future posture sequences given historical posture sequences. In the present invention, only the skeleton point information in the human motion data is retained.

[0033] Mark the historical motion sequence as ,in Respectively represent the length of the historical sequence, the number of skeleton points of a single human posture, and the representation dimension of a single joint point; mark the future sequence as , Indicates the length of the prediction sequence.

[0034] In step 2, step 201 includes: using a Transformer-based backbone model to extract temporal features and spatial features; the feature extraction process of the historical motion sequence is divided into two parts, namely, extracting temporal features and spatial features of the historical motion sequence respectively; both parts adopt a Transformer-based network structure.

[0035] Step 2011: Temporal feature extraction The extraction of temporal features mainly relies on the self-attention mechanism in the original Transformer, which updates its feature representation by considering the correlation between the same joint point in different time frames. In this process, the query, key, and value are all represented by corresponding matrices.

[0036] The value V is represented by a series of high-dimensional features indexed by all keys K. The similarity between the query Q and the key K is calculated by the scaled dot product attention mechanism. The similarity is proportional to the numerical value, and a higher attention weight indicates a higher similarity between the query Q and the key K. The process can be expressed as:

[0037] In the formula, represents the query matrix, represents the key matrix, represents the value matrix, M represents the mask matrix; the present invention introduces the mask matrix into the self-attention mechanism , to prevent the leakage of future information. By using the mask mechanism of the lower triangular matrix, future information is effectively prevented from affecting the human motion reasoning of the current step. The main purpose of introducing the mask mechanism is to ensure that the human motion information in the future steps will not participate in the current reasoning process.

[0038] In addition, the multi-head attention mechanism in the original Transformer mechanism is also adopted to project the joint embedding information of different heads into the subspace , to aggregate information from different angles. Summary of temporal features for each human joint ,Right now , the calculation formula is as follows:

[0039] in, Represents the splicing weight matrix. The calculation result contains the time information of human motion and represents the degree of attention of the current joint to the previous joint.

[0040] Step 2012: Spatial feature extraction For the spatial feature extraction of historical motion sequences, we mainly focus on the interdependence between different joints at the same time step. By modeling the relationship between each joint on the same frame, we use Transformer to extract the spatial features of human joints. The principle is similar to that of temporal feature extraction, and it also uses the multi-head attention mechanism. Figure 3 As shown in , an N-layer Transformer mechanism is used to extract spatiotemporal features.

[0041] Given a human pose embedding and the weight matrix , aggregated at the moment through the multi-head attention mechanism All joint information under, so as to calculate the summary of spatial features , The calculation process is as follows:

[0042] In the formula, , , Represent the query matrix, key matrix, and value matrix respectively. represents the weight matrix.

[0043] Step 202: Interaction of temporal features and spatial features; like Figure 3 As shown, the L layer is used to interact with the extracted features based on the Transformer interactive attention mechanism, including: The self-attention mechanism is used to exchange information between the patch tokens of one feature extraction branch and another feature extraction branch; after obtaining information from the other feature extraction branch, the updated patch tokens are brought back to its own branch through the next spatiotemporal Transformer module, so that the information learned from the other feature extraction branch can be passed back to its own branch to enhance the spatiotemporal representation capability.

[0044] Space information mark With time information sign After dimensional alignment, the query vector Q is obtained by projection. and A cross transformation is performed between them to obtain the key vector K and the value vector V. After the query vector Q is multiplied by the key vector K and the value vector V, the dimension is aligned. After splicing, the spliced ​​features are back-projected and then combined with the spatial information block After splicing, we finally get the consistent spatial features after interaction.

[0045] Specifically: For the task of human motion prediction, an effective method of spatial-temporal branch interaction helps to capture spatial-temporal consistency. Therefore, the present invention explores the interaction between the temporal branch and the spatial branch to promote the feature association between the two branches, designs a temporal-spatial feature interaction attention mechanism, and adds an additional Msg token to the spatial and temporal branches on the basis of the traditional Transformer. The way of spatial interaction with time is consistent with the way of temporal interaction with space, and the interaction of the spatial branch is used as an example; Specifically, in the spatial branch, the "information collection" process is first used to obtain the Msg token (information token) containing the dynamic information of the body joints. Since the Msg token contains the information of all tokens in its branch, it can use the self-attention mechanism to exchange information with the patch token of another branch. This step is called "information interaction". After obtaining information from another branch, the updated Msg token will bring this information back to its own branch through the next spatiotemporal Transformer module, so that the information learned from another branch can be passed back to its own branch to enhance the spatiotemporal representation capability. This process is called "information return". Figure 3 As shown in , an M-layer Transformer mechanism is used to return spatiotemporal features.

[0046] The specific process is as follows:

[0047] in, represents the projection function for dimension alignment. Then, in and Perform cross conversion between them to obtain information of each branch.

[0048] In addition, the present invention introduces a multi-head mechanism in the interactive attention module to project the human motion representation of each head from different angles. Layer normalization and residual connection are also used after each module to improve the stability and performance of the model.

[0049] In summary, the Transformer-based temporal-spatial interactive attention mechanism is expressed as:

[0050] in, and They are the dimension-aligned projection and back-projection functions respectively. Represents the output of the spatial branch in the interactive attention mechanism. The process of the temporal branch is the same as above. After the above process, consistent temporal and spatial features are obtained.

[0051] Step 203: Combine temporal and spatial features for prediction; The interactive time features and spatial features are fused by addition and normalized to obtain the prediction results. The joint prediction process is as follows:

[0052] In the formula, For normalization operation, and They represent spatial features and temporal features respectively, and the final prediction is obtained by adding and normalizing the features in the two directions.

[0053] Step 204: Model training process; The present invention adopts the Adam optimizer to train the Transformer model based on the spatiotemporal interactive attention mechanism and updates the parameters of the model through back propagation.

[0054] In the network learning process, the original Transformer is used as the backbone network of the spatial and temporal branches, and 8 attention heads are configured. The parameters in the network design, the number of feature extractions N, feature interaction L, and feature return M, are set to 1, 1, and 1 respectively.

[0055] The proposed overall algorithm is implemented in the PyTorch framework of Python 3.6. At the same time, to ensure the convergence of the method, the Adam optimizer is used for model training. The initial learning rate is set to , and decays to 98% of the original value every 10 iterations. The method was trained for 300 epochs, with a batch size of 32 for both training and evaluation.

[0056] Step 205: Model verification process; After each training epoch, the validation set was used for verification, and the accuracy was used to evaluate the effect of the model. Finally, the method provided by the present invention achieved a prediction accuracy of 94.65% on the dataset used.

[0057] like Figure 4 As shown in Figure 1, the first row represents the true value of the action sequence, and the second row represents the action prediction value. Among them, the first 6 columns represent the observed action sequence.

[0058] It should be noted that the human motion sequence prediction method provided by the present invention can be applied in autonomous driving and robot scenarios. By predicting the motion trajectory of pedestrians or other objects, the system can plan the path in advance to avoid collisions. It can also be applied in virtual reality (VR) and augmented reality (AR). Human motion prediction makes the movements of virtual characters more realistic and improves the user's sense of immersion. In addition, it can also be applied in medical rehabilitation scenarios. By predicting the patient's movements, it can help formulate personalized rehabilitation plans and assist rehabilitation training.

[0059] Embodiment 2 This embodiment discloses a human motion prediction system based on a spatiotemporal interactive attention mechanism, including: The data acquisition and preprocessing module is configured to: acquire a human body historical motion data set and perform preprocessing to obtain a historical motion sequence; The joint prediction module is configured to: after position encoding the acquired historical motion sequence, input it into the Transformer model based on the spatiotemporal interactive attention mechanism, firstly use the temporal feature extraction branch and the spatial feature extraction branch to extract temporal features and spatial features respectively, then use the interactive attention mechanism to interact with the extracted features, and finally obtain the human motion sequence prediction result by combining the interactive temporal features and spatial features; The interactive attention mechanism is used to interact with the extracted features, including: using the self-attention mechanism to exchange information between a feature extraction branch and the patch tokens of another feature extraction branch; after obtaining information from the other feature extraction branch, the updated patch tokens are brought back to the own branch through the next spatiotemporal Transformer module, so as to pass the information learned from the other feature extraction branch back to the own branch, thereby enhancing the spatiotemporal representation capability.

[0060] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0061] Embodiment 3 Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned method for predicting human motion based on the spatiotemporal interactive attention mechanism are completed.

[0062] Embodiment 4 Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned human motion prediction method based on the spatiotemporal interactive attention mechanism are completed.

[0063] Embodiment 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned human motion prediction method based on the spatiotemporal interactive attention mechanism.

[0064] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0067] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A human motion prediction method based on spatiotemporal interactive attention mechanism, characterized in that: include: Obtain human historical motion data set and preprocess it to obtain historical motion sequence; After the acquired historical motion sequence is position-encoded, it is input into the Transformer model based on the spatiotemporal interactive attention mechanism. First, the temporal feature extraction branch and the spatial feature extraction branch are used to extract temporal features and spatial features respectively. Then, the extracted features are interacted using the interactive attention mechanism. Finally, the human motion sequence prediction result is obtained by combining the interactive temporal features and spatial features. The interactive attention mechanism is used to interact with the extracted features, including: using a self-attention mechanism to exchange information between a patch token of a feature extraction branch and another feature extraction branch; After obtaining information from another feature extraction branch, the updated patch token is brought back to its own branch through the next spatiotemporal Transformer module, so that the information learned from another feature extraction branch can be passed back to its own branch to enhance the spatiotemporal representation capability.

2. A human motion prediction method based on spatiotemporal interactive attention mechanism as claimed in claim 1, characterized in that: The processing process of the Transformer model based on the spatiotemporal interactive attention mechanism includes: using N layers based on the Transformer attention mechanism to extract spatiotemporal features, and distinguishing the spatial features and the temporal features into two parts of feature information marks respectively and feature blocks , mark the feature information in the space branch Characteristic information flags in time branches After dimensional alignment, the query vector Q is obtained by projection, and the spatial feature information mark and time feature blocks A cross conversion is performed between them to obtain the key vector K and the value vector V; after the query vector Q is multiplied by the key vector K and the value vector V, the feature information mark in the spatial branch after the dimension alignment is obtained. After completing the time interaction features, the spliced ​​features are back-projected and then combined with the spatial feature blocks. Splicing is performed to obtain the characteristics after information return; the consistent spatial characteristics after time-space interaction.

3. The method for predicting human motion based on spatiotemporal interactive attention mechanism according to claim 1, characterized in that: The temporal feature extraction branch is used to extract temporal features, including: using the self-attention mechanism in the Transformer to update its feature representation by considering the correlation between the same joint point in different time frames; in the calculation process of the self-attention mechanism, a mask matrix is ​​introduced to prevent future information leakage.

4. The method for predicting human motion based on spatiotemporal interactive attention mechanism according to claim 1, characterized in that: The spatial feature extraction branch is used to extract spatial features, including: given a human posture embedding and weight matrix, using a multi-head attention mechanism to aggregate at time All joint information under the , get the spatial features.

5. The method for predicting human motion based on spatiotemporal interactive attention mechanism as claimed in claim 1, characterized in that: The method of obtaining the human motion sequence prediction result by combining the time features and the space features after the interaction includes: fusing the time features and the space features after the interaction by adding them together, and then obtaining the prediction result by normalization.

6. The method for predicting human motion based on spatiotemporal interactive attention mechanism according to claim 1, characterized in that: The Adam optimizer is used to train the Transformer model based on the spatiotemporal interactive attention mechanism, and the parameters of the model are updated through back propagation.

7. A human motion prediction system based on spatiotemporal interactive attention mechanism, characterized in that: include: The data acquisition and preprocessing module is configured to: acquire a human body historical motion data set and perform preprocessing to obtain a historical motion sequence; The joint prediction module is configured to: after position encoding the acquired historical motion sequence, input it into the Transformer model based on the spatiotemporal interactive attention mechanism, firstly use the temporal feature extraction branch and the spatial feature extraction branch to extract temporal features and spatial features respectively, then use the interactive attention mechanism to interact with the extracted features, and finally obtain the human motion sequence prediction result by combining the interactive temporal features and spatial features; The interactive attention mechanism is used to interact with the extracted features, including: using a self-attention mechanism to exchange information between a patch token of a feature extraction branch and another feature extraction branch; After obtaining information from another feature extraction branch, the updated patch token is brought back to its own branch through the next spatiotemporal Transformer module, so that the information learned from another feature extraction branch can be passed back to its own branch to enhance the spatiotemporal representation capability.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of a human motion prediction method based on a spatiotemporal interactive attention mechanism as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of a human motion prediction method based on a spatiotemporal interactive attention mechanism as described in any one of claims 1-6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of a human motion prediction method based on a spatiotemporal interactive attention mechanism as described in any one of claims 1-6 are implemented.

Images