A human action prediction method and system based on space-time interaction attention mechanism

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a human action prediction method and system based on a space-time interaction attention mechanism. The method comprises the following steps: obtaining a human history motion dataset and preprocessing the same to obtain a history motion sequence, inputting the history motion sequence into a Transformer model based on a space-time interaction attention mechanism, first extracting time features and space features by using a feature extraction branch, then interacting the extracted features by using an interaction attention mechanism, and finally obtaining a human motion sequence prediction result by combining the interacted time features and space features. The present application can learn the consistency and interaction information of 3D human motion space and time features through the interaction attention mechanism, and can produce more realistic 3D human motion in long-time modeling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a human motion prediction method and system based on a spatio-temporal interaction attention mechanism. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Human motion prediction refers to predicting future motion trajectories or poses based on existing motion data and specific conditions. This process involves using computer vision, deep learning, physical simulation, etc. to extract features from past motion patterns and predict the likelihood of the next or a series of consecutive actions of a person. Human motion prediction is widely used in many fields. In sports training, it can help coaches and athletes analyze the rationality of actions and optimize training plans. In human-computer interaction, by predicting the user's actions, the device can react in advance, improving the naturalness and fluency of interaction. In the field of autonomous driving and robotics, by predicting the motion trajectory of pedestrians or other objects, the system can plan the path in advance to avoid collisions. In virtual reality (VR) and augmented reality (AR), human motion prediction makes the actions of virtual characters more realistic, improving the user's sense of immersion. In addition, in the medical rehabilitation field, by predicting the patient's motion, it can help develop personalized rehabilitation plans and assist in rehabilitation training.

[0004] Deep learning methods have significant advantages in human motion prediction tasks, improving the accuracy, flexibility and application range of prediction. First, deep learning can automatically extract high-dimensional features from a large amount of data, independent of hand-designed feature extraction methods. This allows the model to more accurately capture complex motion patterns and temporal relationships, especially when dealing with large-scale, high-dimensional motion data. Second, deep learning models, especially recurrent neural networks (RNN), long short-term memory networks (LSTM) and Transformers, are good at handling time series data and can effectively model the dynamic characteristics of human motion. This allows them to predict continuous motion over a long time span and generate more natural action sequences. In addition, by using deep generative models such as generative adversarial networks (GAN), diverse motion predictions can be generated, considering different possibilities, thereby improving the robustness and generalization ability of the model. Deep learning methods also have good scalability, allowing easy integration of multiple input data types (such as vision, speech, sensor data), thereby improving the accuracy and applicability of prediction. In practical applications such as virtual reality, intelligent monitoring, autonomous driving and robot motion planning, the powerful capabilities of deep learning provide more intelligent and efficient solutions for human motion prediction.

[0005] The Transformer mechanism was proposed by the Google team and is used for machine translation, and has become the main method for natural language processing tasks. The key part of the Transformer is the multi-head self-attention mechanism, which helps to capture long-term dependencies while improving global modeling capabilities. Recent studies have shown that the Transformer network structure significantly improves the performance of various tasks, such as image classification, object detection, and visual question answering systems.

[0006] The Transformer model performs well in extracting spatio-temporal features. Through the self-attention mechanism, the Transformer can simultaneously focus on the relationships between different positions in the sequence, capturing dependencies in the temporal dimension. At the same time, the Transformer can also utilize its multi-head attention mechanism to analyze input data from different angles, obtaining more rich spatial features. When processing spatio-temporal data, the temporal branch of the Transformer is responsible for capturing the dynamic changes between different time frames of actions or events, while the spatial branch focuses on the relationships between objects or body joints in different spatial positions. This dual-branch structure allows the Transformer to effectively integrate temporal and spatial information, providing more accurate and comprehensive feature representations for tasks, and is widely used in human motion prediction, video analysis, and other fields.

[0007] Although the Transformer-based methods have achieved remarkable results, these methods do not have any operations to enhance the continuity between the two branches when processing the interaction of temporal and spatial information, except for the concatenation of spatio-temporal information at the end of the model. Existing methods ignore the consistency of spatio-temporal information, which is an important clue when dealing with time-varying data. SUMMARY

[0008] To solve the problems of the prior art, the present application provides a human action prediction method, system, electronic device, computer readable storage medium and computer program product based on a spatio-temporal interaction attention mechanism,

[0009] To achieve the above-mentioned purpose, one or more embodiments of the present application provide the following technical solutions:

[0010] In a first aspect, the present application provides a human action prediction method based on a spatio-temporal interaction attention mechanism, comprising:

[0011] obtaining a human historical motion dataset and preprocessing to obtain a historical motion sequence;

[0012] The obtained historical motion sequence is positionally encoded and input into a Transformer model based on a space-time interaction attention mechanism. First, time feature extraction and space feature extraction are performed by using a time feature extraction branch and a space feature extraction branch respectively. Then, the extracted features are interacted by using an interaction attention mechanism. Finally, the human motion sequence prediction result is obtained by combining the interacted time features and space features.

[0013] The interaction of the extracted features by using the interaction attention mechanism includes: exchanging information between patch tokens of one feature extraction branch and another feature extraction branch by using a self-attention mechanism; after obtaining information from another feature extraction branch, the updated patch tokens are brought back to the branch itself through a next space-time Transformer module, so as to transmit the information learned from another feature extraction branch back to the branch itself and enhance the space-time representation capability.

[0014] Further, the processing process of the Transformer model based on the space-time interaction attention mechanism includes: extracting space-time features by using an N-layer Transformer attention mechanism, and dividing the space features and the time features into two parts of feature information markers and feature blocks After the feature information markers in the space branch and the feature information markers in the time branch are dimensionally aligned and projected, a query vector Q is obtained, and cross conversion is performed between the space feature information markers and the time feature blocks to obtain a key vector K and a value vector V; after the query vector Q is multiplied by the key vector K and the value vector V, the feature information markers in the space branch after dimensional alignment are spliced; after the time interaction features are completed, the spliced features are de-projected and then spliced with the space feature blocks , so as to obtain the features after information return; the space features after space-time interaction have consistency.

[0015] Further, the time feature extraction by using the time feature extraction branch includes: updating the feature representation of the same joint node by considering the correlation between different time frames by using the self-attention mechanism in the Transformer; during the calculation process of the self-attention mechanism, a mask matrix is introduced to prevent future information leakage.

[0016] Further, the space feature extraction by using the space feature extraction branch includes: given a human pose embedding and a weight matrix, aggregating all joint information at time by using a multi-head attention mechanism to obtain the space features.

[0017] Further, the human motion sequence prediction result obtained by combining the time feature and the space feature after interaction comprises: the time feature and the space feature after interaction are fused by addition, and the prediction result is obtained after normalization.

[0018] Further, the Adam optimizer is used to train the Transformer model based on the spatiotemporal interaction attention mechanism, and the parameters of the model are updated through back propagation.

[0019] In the second aspect, the present application provides a human action prediction system based on a spatiotemporal interaction attention mechanism, comprising:

[0020] The data acquisition and preprocessing module is configured to acquire a human historical motion dataset and preprocess it to obtain a historical motion sequence.

[0021] The joint prediction module is configured to input the obtained historical motion sequence after position encoding into the Transformer model based on the spatiotemporal interaction attention mechanism, first use the time feature extraction branch and the space feature extraction branch to extract the time feature and the space feature respectively, then use the interaction attention mechanism to interact with the extracted features, and finally obtain the human motion sequence prediction result by combining the time feature and the space feature after interaction.

[0022] The interaction of the extracted features by the interaction attention mechanism comprises: using the self-attention mechanism to exchange information between the patch tokens of one feature extraction branch and another feature extraction branch; after obtaining information from another feature extraction branch, the updated patch token is brought back to the branch itself through the next spatiotemporal Transformer module, so as to transmit the information learned from another feature extraction branch back to the branch itself, and enhance the spatiotemporal representation ability.

[0023] In the third aspect, the present application provides an electronic device.

[0024] An electronic device comprises a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the above-mentioned human action prediction method based on the spatiotemporal interaction attention mechanism.

[0025] In the fourth aspect, the present application provides a computer readable storage medium.

[0026] A computer readable storage medium has a computer program / instruction stored thereon, and the computer program / instruction is executed by a processor to realize the steps of the above-mentioned human action prediction method based on the spatiotemporal interaction attention mechanism.

[0027] In a fifth aspect, the present application provides a computer program product;

[0028] A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the above-mentioned human action prediction method based on spatio-temporal interaction attention mechanism.

[0029] The above one or more technical solutions have the following beneficial effects:

[0030] (1) The present application proposes a Transformer-based spatio-temporal attention mechanism interaction method to effectively predict 3D human motion. This method divides the human action sequence information into time branches and space branches, and uses a Transformer-based attention method to extract features of the spatial and temporal branches of human action, aiming to capture the global information of human action.

[0031] (2) The present application designs a Transformer-based spatio-temporal interaction attention mechanism method to learn the consistency and interaction information of 3D human motion spatial and temporal features. This spatio-temporal interaction attention mechanism method is divided into three stages, from Transformer-based spatio-temporal information aggregation to spatio-temporal information interaction, and finally to spatio-temporal information returning to the respective spatio-temporal branch. Through this attention mechanism, the method designed by the present application can produce more realistic 3D human motion in long-time modeling.

[0032] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which form a part of this description, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of the application, and their

[0034] Figure 1 A flow chart of a human action prediction method based on spatio-temporal interaction attention mechanism for the first embodiment.

[0035] Figure 2 A Transformer model structure diagram based on spatio-temporal interaction attention mechanism for the first embodiment.

[0036] Figure 3 An interaction attention mechanism flow chart for the first embodiment;

[0037] Figure 4 A prediction result diagram for the first embodiment. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0039] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the application will be limited only by the appended claims. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. Unless otherwise required by context, singular terms shall include pluralities and vice versa. Plural elements can be separated by a hyphenated form of the element name, for example, "multiple components", unless in the context a singular sense is intended.

[0040] The embodiments in the application and the features in the embodiments can be combined with each other in the case of no conflict.

[0041] Embodiment one

[0042] As shown in Figure 1 , Figure 2 The embodiment discloses a human action prediction method based on a space-time interaction attention mechanism, which comprises the following steps:

[0043] Step 1: Obtain a human historical motion dataset and perform data preprocessing to obtain a historical motion sequence.

[0044] Step 2: After position coding of the obtained historical motion sequence, input it into a Transformer model based on a space-time interaction attention mechanism. First, use a time feature extraction branch and a space feature extraction branch to extract time features and space features respectively. Then, use an interaction attention mechanism to interact the extracted time features and space features. Finally, use the interacted time features and space features to jointly predict a human motion sequence.

[0045] In step 1, step 101: Obtain a human historical motion dataset.

[0046] The application collects a public human motion dataset, which is composed of 3.6 million human poses and corresponding images captured by a high-speed motion capture system. The high-speed motion capture system uses 4 high-resolution progressive scan cameras to obtain 50 Hz video data. The dataset contains the activities of 11 subjects in 17 scenarios such as discussion, smoking, taking pictures, and making phone calls, and provides accurate 3D joint positions and high-resolution videos.

[0047] The data set in the application is divided into training set and validation set, as shown in Table 1, the application uses No. 1, No. 6, No. 7, No. 8 and No. 9 subjects for training, and No. 5 and No. 11 subjects for testing.

[0048] Table 1 Training and validation sample number of human motion data set

[0049]

[0050] Step 102: data preprocessing;

[0051] The purpose of human motion prediction based on skeleton is to predict future pose sequence given historical pose sequence. In the application, only the skeleton point information in human motion data is retained.

[0052] The historical motion sequence is marked as , wherein respectively represent the length of the historical sequence, the number of skeleton points of a single pose of the human body and the representation dimension of a single joint point; the future sequence is marked as , , wherein represents the length of the predicted sequence.

[0053] In step 2, step 201: time feature and space feature are extracted by using a backbone model based on Transformer; the feature extraction process of the historical motion sequence is divided into two parts, i.e. time feature and space feature of the historical motion sequence are extracted respectively; both parts adopt a network structure based on Transformer.

[0054] Step 2011: time feature extraction

[0055] For the extraction of time feature, mainly rely on the self-attention mechanism in the original Transformer, by considering the correlation between the same joint point in different time frames to update its feature representation. In this process, the query (Query), key (Key) and value (Value) are represented by the corresponding matrix.

[0056] The value V is a series of high-dimensional feature representations indexed by all keys K, and the similarity between the query Q and the key K is calculated by the scaling dot product attention mechanism. The similarity is proportional to the numerical size, and the higher attention weight represents that the similarity between the query Q and the key K is higher. The process can be represented as:

[0057]

[0058] In the formula, , wherein represents the query matrix, , represents the key matrix, The value matrix is represented by M, and the mask matrix is represented by M; the mask matrix is introduced in the self-attention mechanism , to prevent the leakage of future information. By using the mask mechanism of the lower triangular matrix, the influence of future information on the inference of human motion at the current step is effectively prevented. The main purpose of introducing the mask mechanism is to ensure that the human motion information in the future step does not participate in the current inference process.

[0059] In addition, the multi-head attention mechanism in the original Transformer mechanism is also adopted, and the joint embedding information of different heads is projected into a subspace to aggregate information from different angles. The summary of each human joint about the time feature , is calculated as follows:

[0060]

[0061] wherein, denotes the splicing weight matrix. The calculation result contains the time information of human motion, and represents the attention degree of the current joint to the previous joint.

[0062] Step 2012: spatial feature extraction

[0063] The spatial feature extraction is performed on the historical motion sequence, mainly focusing on the mutual dependence relationship between different joints at the same time step. By modeling the relationship of each joint on the same frame, the spatial feature of the human joint is extracted by using the Transformer, and the principle is similar to the time feature extraction, which also uses the multi-head attention mechanism. At the same time, as shown in the accompanying Figure 3 , N-layer Transformer mechanism is used for spatio-temporal feature extraction.

[0064] Given a human pose embedding and a weight matrix , all joint information at time is aggregated by using the multi-head attention mechanism, so as to calculate the summary of the spatial feature , The calculation process of

[0065]

[0066] In the formula, , , denote the query matrix, the key matrix, and the value matrix respectively, denotes the weight matrix.

[0067] Step 202: interaction between time feature and spatial feature;

[0068] As shown in Figure 3 , the extracted features are interacted using an L-layer Transformer interaction attention mechanism, including:

[0069] The self-attention mechanism is used to exchange information between the patch tokens of one feature extraction branch and another feature extraction branch. After obtaining information from another feature extraction branch, the updated patch token is brought back to the branch itself through the next spatio-temporal Transformer module, so as to pass the information learned from another branch back to the branch itself and enhance the spatio-temporal representation capability.

[0070] The spatial information token is dimensionally aligned with the temporal information token , and the query vector Q is obtained through projection. The cross conversion is performed between to obtain the key vector K and the value vector V. After the query vector Q is multiplied by the key vector K and the value vector V, it is spliced with the dimensionally aligned , and the spliced feature is spliced with the spatial information block after inverse projection, and finally the spatial feature with consistency after interaction is obtained.

[0071] Specifically, for the human action prediction task, an effective spatio-temporal branch interaction method helps to capture the spatio-temporal consistency. Therefore, the present application explores the interaction mode of the time branch and the space branch to promote the feature correlation between the two branches, and designs a time-space feature interaction attention mechanism. On the basis of the traditional Transformer, additional Msg tokens are added in the space and time branches. The way of space interaction time is consistent with the way of time interaction space, and the space branch interaction is explained as follows:

[0072] Specifically, in the space branch, the Msg token containing the dynamic information of the human joint in the branch is first obtained through the "information collection" process. Since the Msg token contains the information of all tokens in the branch, it can exchange information with the patch token of another branch using the self-attention mechanism. This step is called "information interaction". After obtaining information from another branch, the updated Msg token will bring these information back to its own branch through the next spatio-temporal Transformer module, so as to pass the information learned from another branch back to the branch itself and enhance the spatio-temporal representation capability. This process is called "information return". As shown in Figure 3 , M-layer Transformer mechanism is used for spatio-temporal feature return.

[0073] The specific process is shown as follows:

[0074]

[0075] wherein, denotes a projection function for dimension alignment. and The cross conversion is performed between and to obtain the information of each branch.

[0076] In addition, the application introduces a multi-head mechanism (Multi-Head Attention) in the interactive attention module to project the human motion representation of each head from different angles. Layer normalization (Layer Normalization) and residual connection are also used after each module to improve the stability and performance of the model.

[0077] In summary, the time-space interactive attention mechanism based on the Transformer is represented as:

[0078]

[0079] wherein, and are projection function and inverse projection function for dimension alignment, respectively. denotes the output of the spatial branch in the interactive attention mechanism. The process of the time branch is the same as above. After the above process, the time and space features with consistency are obtained.

[0080] Step 203: joint time-space features for prediction;

[0081] The time feature and the space feature after the interaction are fused by adding, and the prediction result is obtained after normalization. The process of joint prediction is as follows:

[0082]

[0083] In the formula, is a normalization operation, and denote the space feature and the time feature, respectively. The features in two directions are fused by adding and normalized to obtain the final prediction.

[0084] Step 204: training process of the model;

[0085] The Adam optimizer is used to train the Transformer model based on the space-time interactive attention mechanism, and the parameters of the model are updated through back propagation.

[0086] In the network learning process, the original Transformer is used as the backbone network of the space and time branch, and 8 attention heads are configured. The parameter feature extraction quantity N, feature interaction L, and feature return M in the network design are set to 1, 1, and 1 respectively.

[0087] The overall algorithm proposed in the application is implemented in the PyTorch framework of Python 3.6. At the same time, in order to ensure the convergence of the method, the Adam optimizer is used for model training. The initial learning rate is set to , and is attenuated to 98% of the original every 10 iterations. The method is trained for 300 epochs, and the batch size for training and evaluation is 32.

[0088] Step 205: verification process of the model;

[0089] After each training epoch, the validation set is used for verification, and the accuracy is used to evaluate the effect of the model. Finally, the method provided by the application achieves a prediction accuracy of 94.65% on the used data set.

[0090] As shown in Figure 4 , the first row represents the true value of the action sequence, and the second row represents the action prediction value. Among them, the first 6 rows represent the observed action sequence.

[0091] It should be noted that the human action sequence prediction method provided by the application can be applied in automatic driving and robot scenarios. By predicting the motion trajectory of pedestrians or other objects, the system can plan the path in advance to avoid collision. It can also be applied in virtual reality (VR) and augmented reality (AR). Human motion prediction makes the actions of virtual characters more realistic and improves the user's immersion. In addition, it can also be applied in medical rehabilitation scenarios. By predicting the patient's movement, it can help develop personalized rehabilitation plans and assist in rehabilitation training.

[0092] Embodiment Two

[0093] The embodiment discloses a human action prediction system based on a space-time interaction attention mechanism, comprising:

[0094] The data acquisition and preprocessing module is configured to acquire a human historical motion dataset and preprocess it to obtain a historical motion sequence.

[0095] The joint prediction module is configured to: after position coding of the acquired historical motion sequence, input the position coded historical motion sequence into a Transformer model based on a space-time interaction attention mechanism, first utilize a time feature extraction branch and a space feature extraction branch to respectively extract time features and space features, then utilize the interaction attention mechanism to interact the extracted features, and finally obtain a human motion sequence prediction result by jointly interacting the time features and the space features.

[0096] The utilizing the interaction attention mechanism to interact the extracted features comprises: utilizing a self-attention mechanism to exchange information between patch tokens of one feature extraction branch and another feature extraction branch; after obtaining information from another feature extraction branch, the updated patch tokens are brought back to the branch itself through a next space-time Transformer module, so as to transmit the information learned from another feature extraction branch back to the branch itself and enhance the space-time representation capability.

[0097] It should be noted that the above modules correspond to the steps in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules as part of the system can be executed in a computer system such as a group of computer executable instructions.

[0098] Embodiment Three

[0099] Embodiment Three of the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are executed by the processor, the steps of the above-mentioned human action prediction method based on a space-time interaction attention mechanism are completed.

[0100] Embodiment Four

[0101] Embodiment Four of the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps of the above-mentioned human action prediction method based on a space-time interaction attention mechanism are completed.

[0102] Embodiment Five

[0103] Embodiment Five of the present application provides a computer program product comprising computer programs / instructions, which when executed by a processor, implement the steps of the above-mentioned human action prediction method based on a space-time interaction attention mechanism.

[0104] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0105] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0106] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0107] The above description of the various embodiments can have emphasized different features and variations. Details that are not expressly provided herein but are obvious to one skilled in the art are intended to be within the scope of the embodiments. The description of features or aspects within each embodiment should be considered as available for use in other embodiments.

[0108] The preferred embodiments of the present application have been described above with the purpose of enabling not only the best modes contemplated by the inventors of carrying out their application, but also as examples useful in the practice of the present application. The previous description is intended to be illustrative only and not limiting of the present application. Numerous modifications and variations are possible in light of the above teachings without departing from the scope of the present application, which is defined by the following claims.

Claims

1. A human motion prediction method based on spatio-temporal interaction attention mechanism, characterized in that, The method comprises the following steps: acquire a human history motion dataset and preprocess it to obtain a history motion sequence; input the acquired history motion sequence into a Transformer model based on a spatiotemporal interaction attention mechanism after position encoding, first extract time features and space features by using a time feature extraction branch and a space feature extraction branch respectively, then interact the extracted features by using an interaction attention mechanism, and finally obtain a human motion sequence prediction result by combining the interacted time features and space features, including: after the interacted time features and space features are fused by addition, the prediction result is obtained by normalization. The interaction of the extracted features by using the interaction attention mechanism comprises: exchanging information between the patch tokens of one feature extraction branch and the patch tokens of another feature extraction branch by using a self-attention mechanism; after obtaining information from another feature extraction branch, the updated patch tokens are brought back to the branch itself through a next spatiotemporal Transformer module, so as to transmit the information learned from another feature extraction branch back to the branch itself and enhance the spatiotemporal representation capability. The processing process comprises: using N layers of Transformer attention mechanism to extract space-time features, and distinguishing the space features and the time features into two parts of feature information marks and feature blocks After dimension alignment, the feature information marks in the space branch and the feature information marks in the time branch are projected to obtain a query vector Q, and cross conversion is performed between the space feature information marks and the time feature blocks to obtain a key vector K and a value vector V; after the query vector Q is multiplied by the key vector K and the value vector V, the space feature information marks in the dimension alignment are spliced; after the time interaction feature is completed, the spliced feature is de-projected and then spliced with the space feature blocks , thereby obtaining the feature after information return; the space feature after space-time interaction has consistency; The process of joint prediction is shown as follows: wherein, is a normalization operation, and denote spatial and temporal features, respectively, , and are dimension-aligned projection and back-projection functions, respectively; denotes the output of the spatial branch in the cross-attention mechanism, and the process of the temporal branch is the same as above. The time feature extraction by using the time feature extraction branch comprises: updating the feature representation of the same joint node by considering the correlation between different time frames by using the self-attention mechanism in the Transformer; during the calculation of the self-attention mechanism, a mask matrix is introduced to prevent the leakage of future information. 2.The human motion prediction method based on space-time interaction attention mechanism according to claim 1, wherein, The spatial feature extraction branch is used for spatial feature extraction, including: given a human pose embedding and a weight matrix, all joint information at time point t is aggregated by using a multi-head attention mechanism to obtain spatial features. The spatial feature extraction branch is used for spatial feature extraction, including: given a human pose embedding and a weight matrix, all joint information at time point t is aggregated by using a multi-head attention mechanism to obtain spatial features. 3.The human motion prediction method based on space-time interaction attention mechanism according to claim 1, wherein, The Transformer model based on the spatiotemporal interaction attention mechanism is trained by using an Adam optimizer, and the parameters of the model are updated by back propagation.

4. A human motion prediction system based on spatio-temporal interaction attention mechanism, characterized in that, The method comprises the following steps: a data acquisition and preprocessing module configured to acquire a human history motion dataset and preprocess it to obtain a history motion sequence; a joint prediction module configured to input the acquired history motion sequence into a Transformer model based on a spatiotemporal interaction attention mechanism after position encoding, first extract time features and space features by using a time feature extraction branch and a space feature extraction branch respectively, then interact the extracted features by using an interaction attention mechanism, and finally obtain a human motion sequence prediction result by combining the interacted time features and space features, including: after the interacted time features and space features are fused by addition, the prediction result is obtained by normalization. The interaction of the extracted features by using the interaction attention mechanism includes: information exchange between patch tokens of one feature extraction branch and another feature extraction branch by using a self-attention mechanism; after obtaining information from another feature extraction branch, the updated patch tokens are brought back to the branch itself through a next space-time Transformer module, so as to transmit the information learned from another feature extraction branch back to the branch itself and enhance the space-time representation capability, and the processing process includes: extracting space-time features by using N layers of Transformer attention mechanism, and distinguishing the spatial features and the temporal features into two parts of feature information marks and feature blocks After the feature information marks in the spatial branch and the feature information marks in the temporal branch are dimensionally aligned, a query vector Q is obtained through projection, and cross conversion is performed between the spatial feature information marks and the temporal feature blocks to obtain a key vector K and a value vector V; after the query vector Q is multiplied by the key vector K and the value vector V, the feature information marks in the spatial branch after dimension alignment are spliced; after the temporal interaction feature is completed, the spliced feature is de-projected and then spliced with the spatial feature blocks , thereby obtaining the feature after information return; the spatial feature after space-time interaction has consistency; The process of joint prediction is shown as follows: wherein, is a normalization operation, and denote spatial and temporal features, respectively, , and are dimension-aligned projection and back-projection functions, respectively; denotes the output of the spatial branch in the cross-attention mechanism, the process of the temporal branch is the same as above.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 4. The processor executes the computer program to implement the steps of the human action prediction method based on the spatiotemporal interaction attention mechanism according to any one of claims 1-3.

6. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the human action prediction method based on the spatiotemporal interaction attention mechanism according to any one of claims 1-3.

7. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the human action prediction method based on the spatiotemporal interaction attention mechanism according to any one of claims 1-3.

Images