Human motion prediction method and device, intelligent device, and storage medium

Abstract

This invention discloses a method, device, intelligent equipment, and storage medium for predicting human motion. The method includes: acquiring kinematic information of a target human body; performing a pre-defined simplified dynamic calculation on the kinematic information to obtain dynamic information of the target human body; inputting the kinematic and dynamic information into a neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively; and inputting the kinematic and dynamic spatiotemporal features into a neural network decoder to obtain a predicted human motion result for the target human body. Through the human motion prediction method of this invention, kinematic and dynamic information are complementary and coupled in their representation of human motion, comprehensively describing human motion from different perspectives, and enabling more accurate and longer-term predictions of human motion posture over a certain period of time in the future.

Description

Technical Field

[0001] This invention relates to the field of human motion analysis technology, and in particular to a human motion prediction method, device, intelligent equipment, and computer-readable storage medium. Background Technology

[0002] Understanding and predicting human motion is a crucial topic in computer vision. 3D human motion prediction aims to predict the most likely future human pose based on past observed motion data. This task is a fundamental research area in computer science. It plays a key role in a wide range of applications, such as human-computer interaction, motion analysis, autonomous vehicles, and character animation. A common task in this field is to predict a person's most likely future 3D pose by learning a model from 3D pose sequences. The inherent complexity of human behavior makes accurately predicting high-fidelity human motion extremely challenging, and the uncertainty surrounding human pose further complicates long-term predictions.

[0003] However, both traditional probabilistic models and existing deep learning methods only consider the kinematic data of isolated skeletal joints as network input, i.e., only the position or rotation of the joints. These methods ignore the higher-order interactions between skeletal segments (joints), that is, they ignore the dynamic information of the human body. This makes the current human motion prediction results inaccurate and the prediction time short. Summary of the Invention

[0004] The main objective of this invention is to provide a method, device, intelligent equipment, and computer-readable storage medium for predicting human motion, aiming to solve the technical problems that current human motion prediction results are not accurate enough and the prediction time is short.

[0005] To achieve the above objectives, the present invention provides a human motion prediction method, which is applied to a human motion prediction system; the human motion prediction system includes at least: a neural network encoder and a neural network decoder;

[0006] The method includes the following steps:

[0007] Obtain the kinematic information of the target human body, and perform a preset simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body;

[0008] The kinematic information and the dynamic information are input into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively.

[0009] The kinematic spatiotemporal features and the dynamic spatiotemporal features are input into the neural network decoder to obtain the human motion prediction results of the target human body.

[0010] Optionally, the kinematic information includes: joint rotation angles of the human skeletal topology; the dynamic information includes: joint forces between joints;

[0011] The step of performing a pre-defined simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body includes:

[0012] Identify the terminal joints in the human skeletal topology, and determine the joint mass and acceleration of the terminal joints based on the rotation angle of the joints.

[0013] The end joint force at the end joint point is determined based on the joint mass and the acceleration.

[0014] Based on the Newton-Euler iteration rules and the end joint forces, the joint forces between all the joints are calculated by iterating from the end joint inwards.

[0015] Optionally, the step of determining the joint mass of the end joint based on the joint rotation angle includes:

[0016] The three-dimensional distance between the end joint and its corresponding parent joint is determined based on the joint rotation angle.

[0017] The three-dimensional distance is input into a preset joint mass algorithm to obtain the joint mass of the end joint.

[0018] Optionally, the neural network encoder includes a kinematic encoder; the kinematic encoder includes a first spatial Transformer and a first encoder temporal Transformer;

[0019] The step of inputting the kinematic information and the dynamic information into the neural network encoder to obtain the kinematic spatiotemporal features and dynamic spatiotemporal features respectively includes:

[0020] The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features;

[0021] The kinematic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the first encoder time Transformer to obtain kinematic spatiotemporal features.

[0022] Optionally, the neural network encoder includes a dynamic encoder; the dynamic encoder includes a second spatial Transformer and a second encoder temporal Transformer.

[0023] The step of inputting the kinematic information and the dynamic information into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features respectively includes:

[0024] The dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain the dynamic spatial features;

[0025] The dynamic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the second encoder time Transformer to obtain the dynamic spatiotemporal features.

[0026] Optionally, the neural network decoder includes a kinematic decoder and a dynamic decoder; the dynamic decoder includes a third-space Transformer and a first-decoder temporal Transformer.

[0027] The step of inputting the kinematic spatiotemporal features and the dynamic spatiotemporal features into the neural network decoder to obtain the human motion prediction result of the target human body includes:

[0028] The last frame encoded data corresponding to the kinematic information is copied a preset number of times and then input into the third space Transformer to obtain the first query vector.

[0029] The dynamic spatiotemporal features are used as the first key value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector;

[0030] The second query vector and the kinematic spatiotemporal features are input into the kinematic decoder to obtain the human motion prediction result of the target human body.

[0031] Optionally, the kinematic decoder includes a second decoder, the temporal Transformer;

[0032] The step of inputting the second query vector and the kinematic spatiotemporal features into the kinematic decoder to obtain the human motion prediction result of the target human body includes:

[0033] The kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time transformer to obtain the human motion prediction result of the target human body.

[0034] Furthermore, to achieve the above objectives, the present invention also provides a human motion prediction device, the human motion prediction device comprising:

[0035] The dynamics calculation module is used to acquire the kinematic information of the target human body and perform preset simplified dynamics calculations on the kinematic information to obtain the dynamics information of the target human body.

[0036] The feature extraction module is used to input the kinematic information and the dynamic information into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively.

[0037] The prediction output module is used to input the kinematic spatiotemporal features and the dynamic spatiotemporal features into the neural network decoder to obtain the human motion prediction result of the target human body.

[0038] In addition, to achieve the above objectives, the present invention also provides an intelligent device, including a processor, a storage unit, and a human motion prediction program stored on the storage unit and executable by the processor, wherein when the human motion prediction program is executed by the processor, it implements the steps of the human motion prediction method as described above.

[0039] The present invention also provides a computer-readable storage medium storing a human motion prediction program, wherein when the human motion prediction program is executed by a processor, it implements the steps of the human motion prediction method as described above.

[0040] The human motion prediction method in this invention, after acquiring the kinematic information of the target human body, obtains relatively complete dynamic information of the target human body based on simplified dynamic calculations of the kinematic information, which is easy for computer processing. Both the kinematic and dynamic information are input into the neural network encoder of the human motion prediction system to extract kinematic spatiotemporal features and dynamic spatiotemporal features respectively. These two spatiotemporal features are then input into the neural network decoder for coupling. Compared to predicting human motion based solely on kinematic information, this method yields more accurate and longer-term predictions of human motion, i.e., the posture of the human body within a certain future timeframe. This invention integrates the kinematic and dynamic information of the target human body, particularly enhancing the representational ability of human motion features through dynamic information. Dynamic information contains expressive high-order information, and its powerful representational ability has been used to distinguish different movements that are difficult to differentiate in kinematic space. By complementing and coupling the kinematic and dynamic information in the expression of human motion, and comprehensively describing human motion from different perspectives, a more accurate and longer-term prediction of the human body's posture within a certain future timeframe can be made. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the hardware operating environment of the smart device involved in the embodiments of the present invention;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the human motion prediction method of the present invention;

[0043] Figure 3 This is a detailed flowchart of step S10 in an embodiment of the human motion prediction method of the present invention;

[0044] Figure 4 This is a detailed flowchart of step S30 in an embodiment of the human motion prediction method of the present invention;

[0045] Figure 5 This is a schematic diagram of the mass reduction of human joints involved in the human motion prediction method of the present invention.

[0046] Figure 6 This is a schematic diagram of the mechanical relationship between joints involved in the human motion prediction method of the present invention;

[0047] Figure 7 This is a schematic diagram of the neural network framework structure involved in the human motion prediction method of the present invention;

[0048] Figure 8 This is a schematic diagram of the spatiotemporal Transformer structure involved in the human motion prediction method of the present invention;

[0049] Figure 9 This is a schematic diagram of the frame structure of the human motion prediction device of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] This invention provides an intelligent device, which can be a personal computer, a server, etc.

[0053] like Figure 1 As shown, Figure 1 This is a schematic diagram of the hardware operating environment of the smart device involved in the embodiments of the present invention.

[0054] like Figure 1As shown, the intelligent device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a storage unit 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display or an input unit such as a control panel; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The storage unit 1005 may be a high-speed RAM storage unit or a stable storage unit (non-volatile memory), such as a disk storage unit. The storage unit 1005 may also optionally be a storage system independent of the aforementioned processor 1001. The storage unit 1005, as a computer storage medium, may include a human motion prediction program.

[0055] Those skilled in the art will understand that Figure 1 The hardware structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] Continue to refer to Figure 1 , Figure 1 The storage unit 1005, which is a computer-readable storage medium, may include an operating system, a user interface module, a network communication module, and a human motion prediction program.

[0057] exist Figure 1 In this module, the network communication module is mainly used to connect to the server and communicate with it for data; while the processor 1001 can call the human motion prediction program stored in the storage unit 1005 and perform the following operations:

[0058] Obtain the kinematic information of the target human body, and perform a preset simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body;

[0059] The kinematic information and the dynamic information are input into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively.

[0060] The kinematic spatiotemporal features and the dynamic spatiotemporal features are input into the neural network decoder to obtain the human motion prediction results of the target human body.

[0061] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0062] Identify the terminal joints in the human skeletal topology, and determine the joint mass and acceleration of the terminal joints based on the rotation angle of the joints.

[0063] The end joint force at the end joint point is determined based on the joint mass and the acceleration.

[0064] Based on the Newton-Euler iteration rules and the end joint forces, the joint forces between all the joints are calculated by iterating from the end joint inwards.

[0065] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0066] The three-dimensional distance between the end joint and its corresponding parent joint is determined based on the joint rotation angle.

[0067] The three-dimensional distance is input into a preset joint mass algorithm to obtain the joint mass of the end joint.

[0068] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0069] The step of inputting the kinematic information and the dynamic information into the neural network encoder to obtain the kinematic spatiotemporal features and dynamic spatiotemporal features respectively includes:

[0070] The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features;

[0071] The kinematic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the first encoder time Transformer to obtain kinematic spatiotemporal features.

[0072] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0073] The step of inputting the kinematic information and the dynamic information into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features respectively includes:

[0074] The dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain the dynamic spatial features;

[0075] The dynamic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the second encoder time Transformer to obtain the dynamic spatiotemporal features.

[0076] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0077] The last frame encoded data corresponding to the kinematic information is copied a preset number of times and then input into the third space Transformer to obtain the first query vector.

[0078] The dynamic spatiotemporal features are used as the first key value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector;

[0079] The second query vector and the kinematic spatiotemporal features are input into the kinematic decoder to obtain the human motion prediction result of the target human body.

[0080] Furthermore, the processor 1001 can call the human motion prediction program stored in the memory 1005 and also perform the following operations:

[0081] The step of inputting the second query vector and the kinematic spatiotemporal features into the kinematic decoder to obtain the human motion prediction result of the target human body includes:

[0082] The kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time transformer to obtain the human motion prediction result of the target human body.

[0083] To facilitate understanding of the following embodiments of the present invention, the main technical solutions of the present invention will be briefly described below:

[0084] This invention provides a novel non-autoregressive kinematics and dynamics coupled neural network (human motion prediction system). It calculates dynamic information during motion using a simplified human motion reduction algorithm. The neural network is carefully designed to couple kinematic and dynamic information for 3D human motion prediction, achieving better results on various evaluation criteria on publicly available datasets. Furthermore, the employed non-autoregressive decoding strategy fully utilizes both kinematic and dynamic information, enabling fast and accurate motion inference.

[0085] This invention provides a method for predicting human motion.

[0086] Please refer to Figure 2 , Figure 2This is a flowchart illustrating the first embodiment of the human motion prediction method of the present invention; in the first embodiment of the present invention, the human motion prediction method is applied to a human motion prediction system; the human motion prediction system includes at least: a neural network encoder and a neural network decoder;

[0087] The method includes the following steps:

[0088] Step S10: Obtain the kinematic information of the target human body, and perform a preset simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body;

[0089] In this embodiment, the kinematic information of the target human body can be obtained by directly acquiring and utilizing a given dataset on the topological structure of the human skeleton.

[0090] In practical applications, images of the target human body can be captured using image acquisition devices, such as cameras. Kinematic information can then be extracted from these images through image recognition. This kinematic information can include the rotation angles of joints in the human skeletal topology, the spatial positions of joints, or the spatial distances between joints. It is important to clarify that kinematic information refers specifically to the human skeletal topology and the rotation angles, spatial positions, or spatial distances of its joints.

[0091] For the simplified dynamic calculations based on kinematic information, the human body (human skeletal topology) can be divided into five parts according to a pre-defined human structure division rule: the torso, left arm, right arm, left leg, and right leg. The end joint of each part is defined as the end joint point. After obtaining the kinematic information, a pre-defined simplified dynamic calculation can be performed on the end joint points to obtain the force information of the end joint points, which includes joint forces. Then, the dynamic information of each joint point can be solved iteratively from the end joint inward using the Newton-Euler iteration rule (Newton-Euler iteration method). The dynamic information here can include the joint forces between adjacent joint points.

[0092] In this embodiment, the human motion prediction system may further include a point mass model; kinematic information is input into the point mass model to perform preset simplified dynamic calculations to obtain the dynamic information of the target human body.

[0093] This embodiment uses a point mass model instead of a traditional dynamic model to analyze and calculate the dynamic information of the target human body. Traditional dynamic models are constructed using homogeneous columnar models, which are extremely complex for analyzing human dynamics, resulting in lengthy calculations and requiring significant computational power and load. This makes them unsuitable for practical human motion prediction scenarios. Therefore, this embodiment reduces the homogeneous columnar model to a point mass model, transforming the original columnar human skeletal model into a point mass-shaped human skeletal model. The corresponding points are placed at their original joint positions to represent joint points, greatly simplifying the dynamic analysis and calculation between human joint points. This significantly improves the efficiency of obtaining dynamic information from kinematic information, making human motion prediction practical, widely applicable, and feasible.

[0094] To better understand the reduction of the homogeneous columnar model to a point mass model mentioned above, please refer to... Figure 5 ,like Figure 5 As shown, Figure 5 As an example, part (a) on the left is a dynamic model of a three-dimensional object built into a columnar structure, which can be reduced to a mass model of a structure composed of individual mass points on the right. Similarly, the traditional human body structure built into a columnar structure can also be reduced to a human body structure composed of joint points represented by individual mass points, as shown in part (b). Based on this human body structure, simplified dynamic calculations are performed on each joint point from the end joint point inward to obtain the force state of each joint point, thereby determining the joint force between each adjacent joint point.

[0095] Please refer to Figure 3 In one embodiment, the kinematic information includes: joint rotation angles of the human skeletal topology; the dynamic information includes: joint forces between joints;

[0096] Step S10 involves performing a pre-defined simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body, including:

[0097] Step S11: Determine the terminal joints in the human skeletal topology, and determine the joint mass and acceleration of the terminal joints based on the joint rotation angle.

[0098] As mentioned earlier, distal joints can be determined by dividing the human skeletal topology based on preset human structural division rules. That is, the distal joints of each human body component obtained from this division are, as... Figure 5 As shown in (b), each human body part is within its corresponding dashed box, and each end point represents the end joint.

[0099] Please refer to Figure 6 , Figure 6 for Figure 5 (b) Force analysis of each joint point (numbered 1 to 4, 5 is the joint connecting the torso to the left and right legs) of the target human body in the right leg. The end joint point of the right leg can be the ankle joint, and the figure shows joint point number 1.

[0100] To obtain the joint mass of each joint, including the end joint, it is reasonable to assume that the cross-sectional density between two connecting joints is uniformly distributed. Therefore, the joint mass is proportional to the three-dimensional distance between the two joints. The joint mass m of joint j in the k-th component (based on the numbering of the five human body components mentioned above) is... k,j It can be defined as:

[0101]

[0102] Where, x k，j and x k，j+1 Let n be the spatial position of the j-th joint and its parent joint in the k-th part. k This refers to the number of joints in component k. For example, the right leg component has 4 joints (each component can have more or fewer joints depending on actual needs). It should be noted that the parent joint refers to the joint that is inward and adjacent to the currently being calculated joint. For example, the parent joint of the end node numbered 1 is the joint numbered 2, the parent joint of the joint numbered 2 is the joint numbered 3, and so on.

[0103] The mass of any joint can be calculated using the above formula (1).

[0104] Further, the step of determining the joint mass of the end joint based on the joint rotation angle includes:

[0105] Step a: Determine the three-dimensional distance between the end joint and the parent joint corresponding to the end joint based on the joint rotation angle;

[0106] Step b: Input the three-dimensional distance into a preset joint mass algorithm to obtain the joint mass of the end joint.

[0107] In this embodiment, the spatial positions of various joints in the human body, including the distal joints, can be converted from joint rotation angles to corresponding spatial positions using a forward kinematics algorithm. If the kinematic information includes the spatial positions of each joint in the human skeletal topology or the spatial distances between joints (between child and parent joints), then the three-dimensional distances can be obtained directly without conversion, and specific kinematic information can be selected according to actual needs.

[0108] After obtaining the spatial position of the end joint and the spatial position of its corresponding parent node directly or through conversion, the three-dimensional distance between the end joint and its corresponding parent node, i.e., the spatial distance, is obtained. This three-dimensional distance is input into a preset joint mass algorithm, i.e., the above formula (1), and combined with the parameter of the number of joints, the joint mass of the end joint is calculated. Similarly, the joint mass of each other joint can be obtained by calculating the spatial distance between it and its parent joint.

[0109] To obtain the acceleration of each joint, including the terminal joint, it can be obtained based on the following kinematic formula (2).

[0110]

[0111] in, Let be the spatial position of joint j in frames t-1, t, and t+1. It should be noted that the target human body can be sampled by a preset FPS (Frames Per Second), thereby arranging the kinematic information according to the frames in different time sequences. The kinematic information is also known as the kinematic data sequence. The preset FPS is not limited and can be set according to actual needs.

[0112] Step S12: Determine the end joint force of the end joint point based on the joint mass and the acceleration;

[0113] The joint forces at each joint, including the end joint, can be obtained by solving the following mechanical equilibrium formula (3).

[0114] For the mechanical equilibrium equation at joint j:

[0115]

[0116] in It is the joint force of the j-th joint in frame t. It is the acceleration of the j-th joint in frame t, m j Let g be the mass of the j-th joint, and g be the acceleration due to gravity.

[0117] In this embodiment, external forces acting on the human body can be disregarded, thus defining the external forces on the end joints as zero. Therefore, for the end joint forces, after calculating the joint mass and acceleration of the end joint, as follows... Figure 6 As shown, the force information at the end joint includes: gravity m1g, and the end joint force from the parent joint. Based on the resultant force of these two forces It can be seen that the force at the end joint can be obtained by solving for the other two forces.

[0118] Step S13: Based on the Newton-Euler iteration rules and the end joint forces, iterate from the end joint inwards to calculate the joint forces between all the joints.

[0119] For joints other than the end joint, there are not only gravity, joint forces from the parent joint, and the resultant force obtained from acceleration, but also the reaction forces corresponding to child joints (for example, joint 1 corresponding to joint 2 is a child joint of joint 2).

[0120] Therefore, for the convenience and efficiency of calculation, starting from the joint forces of the end joints obtained from the solution, we perform iterative calculations inward based on the Newton-Euler iteration rules to obtain the joint forces between all joints and the overall force situation of each joint. Here, the joint force refers to the force exerted by the parent joint on the joint.

[0121] To better understand the above process, we can see Figure 6 In the process, after calculating the forces on the distal joint 1, the forces on joint 2 are obtained sequentially based on the forces on the distal joint: m2g、 in express The reaction force. Forces at joint 3: m3g、 in express The reaction force. Forces at joint 4: m4g in express The reaction force. Forces at joint 5: m5g, in express The reaction force.

[0122] Through the above embodiment of the present invention, dynamic information can be further obtained on the basis of kinematic information, and the whole calculation process is relatively simple, with better computational efficiency and effect. Thus, dynamic information can be quickly obtained based on kinematic information, and then the two can be coupled to predict more accurate human movement.

[0123] Step S20: Input the kinematic information and the dynamic information into the neural network encoder to obtain the kinematic spatiotemporal features and dynamic spatiotemporal features, respectively;

[0124] Kinematic and dynamic information are input parameters to a neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively. The neural network encoder includes a kinematic encoder and a dynamic encoder. Kinematic information is input to the kinematic encoder to extract kinematic spatiotemporal features, and dynamic information is input to the dynamic encoder to extract dynamic spatiotemporal features. Here, spatiotemporal features refer to time-space characteristics.

[0125] In one embodiment, the neural network encoder includes a kinematic encoder; the kinematic encoder includes a first spatial Transformer and a first encoder temporal Transformer;

[0126] Step S20 includes:

[0127] Step c: The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features;

[0128] Step d: Flatten the kinematic spatial features and combine them with a preset encoder time position embedding function, then input them into the first encoder time Transformer to obtain kinematic spatiotemporal features.

[0129] In another embodiment, the neural network encoder includes a dynamic encoder; the dynamic encoder includes a second spatial Transformer and a second encoder temporal Transformer;

[0130] Step S20 includes:

[0131] Step e: The dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain dynamic spatial features;

[0132] Step f involves flattening the dynamic spatial features and combining them with a preset encoder time-position embedding function before inputting them into the second encoder time Transformer to obtain the dynamic spatiotemporal features.

[0133] Based on the same technical principles, the two embodiments described above will be explained in combination.

[0134] First, it's important to clarify that neural network encoders are all based on spatiotemporal Transformers, which consist of four spatial Transformer layers and four temporal Transformer layers stacked together. The spatial and temporal Transformers share the same structure, comprising a Multi-Head Self-Attention (MHSA) module and a Feed Forward Network (FFN). The MHSA consists of multiple self-attention modules used to adaptively establish relationships between input labels, while the FFN consists of two fully connected (FC) layers used for feature transformation. Residual addition is performed after both the MHSA and FFN modules.

[0135] For a skeleton with 32 joints (or any number of joints depending on the needs), each joint is treated as a token, and a transformation matrix W is used to... s ∈R 32×3 Each label is embedded into a 32-dimensional space, and then embedded with an E at a learnable spatial location. pe1 ∈R 32×32 The summation, that is, the result of the summation, will be used as the input of the spatial Transformer. This process can be represented by the following formula (4).

[0136]

[0137] Where mk, md, and ds represent motion kinematics (mk), motion dynamics (md), and decoder spatial-transformer (ds), respectively. Therefore, W φ This is the embedding matrix, X φ The input is a motion sequence (kinematic information). It should be noted that the spatial Transformer of the neural network encoder and decoder is the same; only the temporal Transformer differs between the encoder and decoder.

[0138] Since the MHSA module has a head count of 8 and an embedding dimension of 32, the spatial features X of the input sequence (kinematic or dynamic information) are ultimately obtained. so ∈R n×32×32 .

[0139] It should also be noted that the spatial location is embedded in E pe1It consists of learnable position embeddings used to distinguish different joints of the skeleton, and it acts on the spatial Transformers of the kinematic encoder, dynamic encoder and decoder respectively.

[0140] For a kinematic encoder, after going through the above steps, since the input is kinematic information, it first goes through the first spatial Transformer to obtain kinematic spatial features.

[0141] For the dynamic encoder, after the above steps, since the input is dynamic information, it first passes through the second-space Transformer to obtain the dynamic space features. The first-space Transformer and the second-space Transformer are consistent neural network structures.

[0142] The encoder temporal Transformer focuses on establishing the temporal relationships between different frames of the input motion sequence; it is also composed of four stacked Transformer blocks. The output X of the aforementioned spatial Transformer... so Flatten it out and turn it into an X so ∈R n ×1024 Simultaneously, the learnable encoder time position is embedded in E. pe2 ∈R n×1024 The result after flattening X so The summation is used as the input to the time Transformer; that is, the input to the time Transformer can be expressed as:

[0143]

[0144] in Flatten the output vector of the spatial Transformer.

[0145] In the Temporal Transformer, the MHSA module has a head size of 8 and an embedding dimension of 1024. The Temporal Transformer follows the same steps as the Spatial Transformer, ultimately obtaining the spatiotemporal features X of the input historical sequence (kinematic or dynamic information). to ∈R n×1024。

[0146] It should also be noted that the encoder time position embedding E pe2 It consists of learnable position embeddings, used to distinguish different frames in the historical sequence, and it acts on the temporal Transformer positions of the kinematic encoder and the dynamic encoder respectively.

[0147] For the kinematic encoder, after obtaining the kinematic spatial features through the first spatial transformer, the kinematic spatial features are input into the first encoder temporal transformer to obtain the kinematic spatiotemporal features.

[0148] For the dynamic encoder, after obtaining the dynamic space features through the second space Transformer, the dynamic space features are input into the second encoder time Transformer to obtain the dynamic spatiotemporal features.

[0149] To briefly summarize the two embodiments above:

[0150] A neural network encoder consists of a kinematic encoder and a dynamic encoder. The kinematic encoder and dynamic encoder are both constructed from identical spatiotemporal Transformers. For a given historical motion sequence, the kinematic information can be expressed using the rotation angle X. mk This means that, firstly, the proposed order reduction dynamics algorithm Φ(·) is used, where Φ(·) represents the overall order reduction operation including formulas (1), (2), and (3), to calculate the dynamic information X of the motion. md , that is, X md =Φ(X mk Rotation angle X mk As input to the kinematic encoder, the kinematic spatiotemporal features are processed by the spatiotemporal Transformer within the kinematic encoder to obtain the kinematic spatiotemporal feature representation. Joint force X md As input to the dynamic encoder, the dynamic encoder's spatiotemporal Transformer processes the data to obtain the spatiotemporal feature representation of the dynamics.

[0151] In the two embodiments of the present invention, based on the above-mentioned neural network encoder structure, it is possible to efficiently, accurately and comprehensively extract the corresponding kinematic spatiotemporal features and dynamic spatiotemporal features respectively based on kinematic information and dynamic information, thereby facilitating the subsequent coupling of the two features and predicting more accurate human motion.

[0152] Step S30: Input the kinematic spatiotemporal features and the dynamic spatiotemporal features into the neural network decoder to obtain the human motion prediction result of the target human body.

[0153] The neural network decoder consists of a kinematic decoder and a dynamic decoder. The kinematic decoder and dynamic decoder can have the same spatial Transformer as the kinematic encoder and dynamic encoder mentioned above, but have different temporal Transformers.

[0154] For the spatial Transformer, it consists of a multi-head self-attention (MHSA) module and feed-forward networks (FFN). For the temporal Transformer, it consists of multiple cross-attention (MHCA) modules used to establish the relationship between input labels and historical information; in this invention, this is used to establish the relationship between joints and rotation angles in kinematic information or joint forces in dynamic information. The FFN consists of two fully connected layers (FC) used for feature transformation. Residual addition is performed after both the MHCA and FFN modules.

[0155] Since predicting future human motion primarily involves determining the last action a person will take, the kinematic data of the last frame is found and determined from the kinematic information, and a certain number of copies are made to form a new kinematic sequence. These multiple copies of the last frame's kinematic data and the spatiotemporal dynamic features are then input into the kinematic decoder to establish the relationship between historical kinematic information (i.e., the acquired kinematic information of the target human body) and the aforementioned new kinematic sequence (the current kinematic sequence). This results in a certain output. The output, along with the kinematic spatiotemporal features output by the kinematic encoder, is then input into the kinematic decoder to establish the relationship between historical kinematic information and the current kinematic sequence, thereby obtaining the undimension-reduced motion prediction result data. This data is then reduced in dimension to obtain the future motion sequence output result, which is the final output of the target human body's motion prediction result.

[0156] Please refer to Figure 4 In one embodiment, the neural network decoder includes a kinematic decoder and a dynamic decoder; the dynamic decoder includes a third spatial Transformer and a first decoder temporal Transformer.

[0157] The step of inputting the kinematic spatiotemporal features and the dynamic spatiotemporal features into the neural network decoder to obtain the human motion prediction result of the target human body includes:

[0158] Step S31: Copy the last frame encoded data corresponding to the kinematic information a preset number of times and input it into the third space Transformer to obtain the first query vector;

[0159] For ease of distinction, the spatial Transformer in the kinematic decoder is referred to as the third spatial Transformer, but it has the same structure (MHSA+FFN) as the first and second spatial Transformers mentioned above. The temporal Transformer in the kinematic encoder, the first decoder temporal Transformer, has a structure of MHCA+FFN, which is different from the MHSA+FFN structure of the temporal Transformers in the various encoders mentioned above.

[0160] Kinematic information can be represented as (x1, x2, x3...x n The kinematic sequence is represented by , where 1 to n represent frames, such as frame 1, frame 2, etc. Correspondingly, the last frame of coded kinematic information is x. n It is copied a predetermined number of m times to obtain a new kinematic sequence (x). n x n ...x n The preset quantity m here can be set according to actual needs.

[0161] The new kinematic sequence (x) n x n ...x n The result obtained by inputting into the third-space Transformer E is embedded after the decoder's time position. pe3 (Constituted from learnable positional embeddings, used to distinguish different frames in the sequence to be predicted, which can be applied to the temporal Transformer position of the dynamics decoder) After flattening, we get The corresponding calculation formula is:

[0162]

[0163] Will The query (Q) vector of the MHCA of the first decoder time-transformer, i.e. the first query vector, is used as the input of the first decoder time-transformer.

[0164] Step S32: The dynamic spatiotemporal features are used as the first key value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector;

[0165] The dynamic spatiotemporal characteristics output by the dynamic encoder As the key (K) and value (V) vector of MHCA, along with the first query vector mentioned above. The data is input into the first decoder, the Time Transformer, where a dynamics decoder consisting of four layers of MHCA+FFN establishes the relationship between historical dynamics information and the current sequence. Its output is then used... This indicates the output of the dynamics decoder. This serves as the query (Q) vector for MHCA in the kinematic decoder, i.e., the second query vector.

[0166] Step S33: Input the second query vector and the kinematic spatiotemporal features into the kinematic decoder to obtain the human motion prediction result of the target human body.

[0167] The second query vector and the aforementioned kinematic spatiotemporal features All are input into the kinematic decoder, including the kinematic spatiotemporal features. As the key (K) and value (V) vectors in the kinematic decoder MHCA, i.e., the second key-value vector, the kinematic decoder ultimately yields the predicted human motion of the target human body.

[0168] Specifically, in one embodiment, the kinematic decoder includes a second decoder, the temporal Transformer;

[0169] The step of inputting the second query vector and the kinematic spatiotemporal features into the kinematic decoder to obtain the human motion prediction result of the target human body includes:

[0170] Step g: The kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time Transformer to obtain the human motion prediction result of the target human body.

[0171] The second decoder temporal Transformer has the same structure as the first decoder temporal Transformer. The kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder temporal Transformer, that is... and The input is fed to the second decoder, the Time Transformer, which uses a kinematic decoder consisting of four layers of MHCA+FFN to establish the relationship between historical dynamics information and the current sequence. Its output is used for... The output of the kinematic decoder is then processed through a fully connected (FC) layer. Perform dimensionality reduction and obtain the motion prediction residual offset Y for the next m frames. res ∈R m×99 It can be represented as (Y1, Y2, Y3...Ym ), the motion prediction residual offset Y res The query input to the decoder is a copy of the last m frames of the historical motion sequence. The sums are used to obtain the future motion sequence output R, which determines the predicted motion result of the target human body.

[0172] The human motion prediction method in this invention, after acquiring the kinematic information of the target human body, obtains relatively complete dynamic information of the target human body based on simplified dynamic calculations of the kinematic information, which is easy for computer processing. Both the kinematic and dynamic information are input into the neural network encoder of the human motion prediction system to extract kinematic spatiotemporal features and dynamic spatiotemporal features respectively. These two spatiotemporal features are then input into the neural network decoder for coupling. Compared to predicting human motion based solely on kinematic information, this method yields more accurate and longer-term predictions of human motion, i.e., the posture of the human body within a certain future timeframe. This invention integrates the kinematic and dynamic information of the target human body, particularly enhancing the representational ability of human motion features through dynamic information. Dynamic information contains expressive high-order information, and its powerful representational ability has been used to distinguish different movements that are difficult to differentiate in kinematic space. By complementing and coupling the kinematic and dynamic information in the expression of human motion, and comprehensively describing human motion from different perspectives, a more accurate and longer-term prediction of the human body's posture within a certain future timeframe can be made.

[0173] Furthermore, since the above embodiments of the present invention involve some relatively low-level and deep neural network model operations, please refer to [the relevant documentation] for further improvement and convenience in understanding the present invention. Figure 7 and Figure 8 .

[0174] for Figure 7 It mainly consists of the basic structure of encoders and decoders for human motion prediction systems, including dynamic encoders, kinematic encoders and decoders (dynamic decoders and kinematic decoders), fully connected layers (FC), and QT (quaternion transformations) layers.

[0175] Obtain the kinematic sequence (x1, x2, x3...x) of the target human body n The kinematic information is used to obtain dynamic information. The kinematic and dynamic information are then input into the kinematic encoder and dynamic encoder respectively via fully connected circuits to extract kinematic and dynamic spatiotemporal features. These features are then input into the corresponding decoders, along with the last frame data x. nData after copying m times (x n x n ...x n The data is input to the corresponding decoder through a fully connected circuit, thus based on three data points: kinematic spatiotemporal features, dynamic spatiotemporal features, and the last frame data x. n Data after copying m times (x n x n ...x n Finally, the QT layer outputs the human motion prediction results (Y1, Y2, Y3...Y...). m ).

[0176] for Figure 8 Temporal Transformer represents the time-based Transformer, and Spatial Transformer represents the spatial Transformer. The spatial Transformer and the temporal Transformer in the encoder have the same structure; the difference lies in the temporal Transformer between the encoder and decoder. For ease of distinction, the K-Temporal Transformer is used as the temporal Transformer in the kinematic decoder, and the D-Temporal Transformer is used as the temporal Transformer in the dynamic decoder.

[0177] for Figure 8 The spatial Transformer in the encoder and the temporal Transformer in the encoder are composed of MHSA and FFN, with the LN representation layer normalized. For Figure 8 The temporal Transformer in the decoder consists of MHCA and FFN.

[0178] Figure 8 In this context, K / D-Encoder stands for kinematic encoder or dynamic encoder; Decoder is the decoder, which includes both kinematic decoder and dynamic decoder.

[0179] L represents the number of iterations, which can be set according to actual needs. PE1, PE2, PE3, and P respectively represent the spatial position embedding E mentioned above. pe1 Encoder time and position embedding E pe2 and decoder time position embedding E pe3 K1 and V1 represent the dynamic spatiotemporal characteristics. K2 and V2 refer to the kinematic spatiotemporal characteristics. Q represents the query vector, including the first and second query vectors mentioned above.

[0180] In one embodiment, the human motion prediction system further includes: a quaternion transformation layer; converting the exponential mapping predicted by the decoder into quaternions, optimizing the system through training in the quaternion space, setting a loss function to constrain the neural network, and defining an evaluation index.

[0181] Specifically, a quaternion transformation layer converts the exponential mapping predicted by the decoder into quaternions. Through training and optimization in the quaternion space, a loss function is set to constrain the neural network. Quaternions can effectively eliminate singularities and discontinuities. To utilize the stable numerical advantages of quaternions, a quaternion transformation layer is designed. Assuming the human body has 32 joints, etc., e t，j Let x be the exponential mapping of joint j in frame t. Then the pose of frame t can be represented as x. t =[e t，1 ;…;e t，j ;…;e t，J For each joint j∈{1,2,...,J}, J=32, its exponent is mapped to e using a quaternion transformation layer. t，j (3D vector) is converted into a 4D vector q. t，j :

[0182]

[0183] Where q t，j q is the quaternion of joint j in frame t. t，j (i) is the i-th element, i∈{1,2,3,4}, and ||·||2 is the L2 norm.

[0184] In one embodiment, the loss function, during training, aims to minimize the difference between the predicted pose and the ground truth in quaternion space. Therefore, a quaternion transformation layer is used to convert the predicted kinematic data into quaternions for loss calculation. The training loss for m-frame motion segments is defined as follows:

[0185]

[0186] Where g represents quaternion transformation operation, y n+t，j For Ground Truth, r n+t，j Let m be the pose to be predicted by the network, n be the number of frames in the sequence to be predicted, J be the total number of joints, t be the t-th frame, and j be the j-th joint. ||·||1 is the L1 norm.

[0187] In one embodiment, the evaluation metric, during the test evaluation phase, removes the quaternion transformation layer from the network and uses the original exponential mapping to represent human pose. Therefore, the prediction performance is evaluated based on the error in the exponential mapping space. Thus, the prediction error for predicting m-frame motion segments is defined as:

[0188]

[0189] Where y n+t, For Ground Truth, r n+t, Let m be the number of frames in the sequence to be predicted, n be the number of frames in the historical sequence, J be the total number of joints, t be the t-th frame, and j be the j-th joint. ‖·‖2 is the L2 norm.

[0190] Through this embodiment of the present invention, the human motion prediction system can be continuously optimized, thereby further improving the accuracy, efficiency, and predictability of human motion prediction during the continuous iterative learning process.

[0191] The testing process and results of the technical solution of this invention:

[0192] The embodiments used Python as the programming language and an Intel(R) i7 8700K CPU (4.8GHz, 8 cores) and 32GB RAM, running on a Windows 10 64-bit system. Two different datasets were used in these embodiments. The Human3.6M dataset consisted of 7 subjects performing 15 activities, with 50FPS motion sequences downsampled by a factor of 2. Data from subject 5 was used for testing, while the other datasets were used for training. Before the experiment, the dynamic information of all samples was pre-calculated, and the dynamic data was filtered using a 6Hz filter. The reason for filtering is that inverse dynamics involves the calculation of derivatives. After filtering, the influence of noise can be greatly eliminated. During testing, the Euclidean distance between the predicted values ​​and the ground truth values ​​was measured using Euler angles, and the test error of the motion under 8 different seeds was reported. For the CMU MoCap dataset, samples of two or more person movements were removed, and only single-person movements were selected. In the experiment, the motion sequences were sampled at a frame rate of 30FPS. The data was preprocessed, and the results were evaluated, just as done on the Human 3.6M dataset. During evaluation, the average distance between 80 sampled seed segments was reported. In the experiments, the input history sequence length n was set to 50, and the predicted sequence length m was set to 25. During training, the batch size and maximum number of training iterations were 32 and 10,000, respectively, and the Adam optimizer with a learning rate of 0.0001 was used to optimize our model. Sequences shorter than 500 milliseconds (ms) were considered short-term predictions, and sequences longer than 500 ms were considered long-term predictions. Therefore, based on 50 previously observed frames, 10 frames (0.4 seconds) of short-term motion and 25 frames (1 second) of long-term motion were predicted. Compared to existing techniques using similar test conditions and environments, this prediction is more accurate and has a longer prediction duration. Furthermore, the accuracy and prediction duration can be further improved with more powerful computers, resulting in more efficient, accurate, and longer-predictable predictions.

[0193] In addition, refer to Figure 9 , Figure 9 This is a schematic diagram of the frame structure of the human motion prediction device of the present invention. The present invention also proposes a human motion prediction device, which includes:

[0194] The dynamic calculation module A10 is used to acquire the kinematic information of the target human body and perform a preset simplified dynamic calculation on the kinematic information to obtain the dynamic information of the target human body.

[0195] The feature extraction module A20 is used to input the kinematic information and the dynamic information into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively.

[0196] The prediction output module A30 is used to input the kinematic spatiotemporal features and the dynamic spatiotemporal features into the neural network decoder to obtain the human motion prediction result of the target human body.

[0197] Optionally, the power calculation module A10 is further configured to:

[0198] Identify the terminal joints in the human skeletal topology, and determine the joint mass and acceleration of the terminal joints based on the rotation angle of the joints.

[0199] The end joint force at the end joint point is determined based on the joint mass and the acceleration.

[0200] Based on the Newton-Euler iteration rules and the end joint forces, the joint forces between all the joints are calculated by iterating from the end joint inwards.

[0201] Optionally, the power calculation module A10 is further configured to:

[0202] The three-dimensional distance between the end joint and its corresponding parent joint is determined based on the joint rotation angle.

[0203] The three-dimensional distance is input into a preset joint mass algorithm to obtain the joint mass of the end joint.

[0204] Optionally, the feature extraction module A20 is further configured to:

[0205] The step of inputting the kinematic information and the dynamic information into the neural network encoder to obtain the kinematic spatiotemporal features and dynamic spatiotemporal features respectively includes:

[0206] The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features;

[0207] The kinematic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the first encoder time Transformer to obtain kinematic spatiotemporal features.

[0208] Optionally, the feature extraction module A20 is further configured to:

[0209] The dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain the dynamic spatial features;

[0210] The dynamic spatial features are flattened and combined with a preset encoder time-position embedding function, then input into the second encoder time Transformer to obtain the dynamic spatiotemporal features.

[0211] Optionally, the prediction output module A30 is further configured to:

[0212] The last frame encoded data corresponding to the kinematic information is copied a preset number of times and then input into the third space Transformer to obtain the first query vector.

[0213] The dynamic spatiotemporal features are used as the first key value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector;

[0214] The second query vector and the kinematic spatiotemporal features are input into the kinematic decoder to obtain the human motion prediction result of the target human body.

[0215] Optionally, the prediction output module A30 is further configured to:

[0216] The kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time transformer to obtain the human motion prediction result of the target human body.

[0217] The specific implementation of the human motion prediction device of the present invention is basically the same as the embodiments of the human motion prediction method described above, and will not be repeated here.

[0218] Furthermore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium of the present invention stores a human motion prediction program, wherein, when executed by a processor, the human motion prediction program implements the steps of the human motion prediction method as described above.

[0219] The method implemented when the human motion prediction program is executed can be referred to in various embodiments of the human motion prediction method of the present invention, and will not be repeated here.

[0220] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0222] These computer program instructions may also be stored in a computer-readable storage unit that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage unit produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0223] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0224] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0225] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0226] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for predicting human motion, characterized in that, The human motion prediction method is applied to a human motion prediction system; the human motion prediction system includes at least: a neural network encoder and a neural network decoder, the neural network encoder including a kinematic encoder and a dynamic encoder, the kinematic encoder including a first spatial Transformer and a first encoder-time Transformer, the dynamic encoder including a second spatial Transformer and a second encoder-time Transformer; the neural network decoder including a kinematic decoder and a dynamic decoder, the dynamic decoder including a third spatial Transformer and a first decoder-time Transformer, the kinematic decoder including a second decoder-time Transformer; the human motion prediction system also includes a quaternion transformation layer; The method includes the following steps: Obtain the kinematic information of the target human body, and perform a preset simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body; The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features; the kinematic spatial features are flattened and combined with a preset encoder temporal position embedding function and then input into the first encoder temporal Transformer to obtain kinematic spatiotemporal features; the dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain dynamic spatial features; the dynamic spatial features are flattened and combined with a preset encoder temporal position embedding function and then input into the second encoder temporal Transformer to obtain dynamic spatiotemporal features; The last frame of encoded data corresponding to the kinematic information is copied a preset number of times and input into the third space Transformer to obtain the first query vector; the dynamic spatiotemporal features are used as the first key-value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector; the kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time Transformer to obtain the initial prediction result; the exponential mapping of the initial prediction result is converted into quaternions through the quaternion transformation layer, and training and optimization are performed in the quaternion space to obtain the human motion prediction result of the target human body.

2. The human motion prediction method as described in claim 1, characterized in that, The kinematic information includes: the rotation angles of the joints in the topology of the human skeleton; the dynamic information includes: the joint forces between the joints. The step of performing a pre-defined simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body includes: Identify the terminal joints in the human skeletal topology, and determine the joint mass and acceleration of the terminal joints based on the rotation angle of the joints. The end joint force at the end joint point is determined based on the joint mass and the acceleration. Based on the Newton-Euler iteration rules and the end joint forces, the joint forces between all the joints are calculated by iterating from the end joint inwards.

3. The human motion prediction method as described in claim 2, characterized in that, The step of determining the joint mass of the end joint based on the joint rotation angle includes: The three-dimensional distance between the end joint and its corresponding parent joint is determined based on the joint rotation angle. The three-dimensional distance is input into a preset joint mass algorithm to obtain the joint mass of the end joint.

4. A human motion prediction device, characterized in that, The human motion prediction device includes: The dynamics calculation module is used to acquire the kinematic information of the target human body and perform preset simplified dynamics calculations on the kinematic information to obtain the dynamics information of the target human body. The feature extraction module is used to input the kinematic information and the dynamic information into the neural network encoder to obtain kinematic spatiotemporal features and dynamic spatiotemporal features, respectively. A prediction output module is used to input the kinematic spatiotemporal features and the dynamic spatiotemporal features into a neural network decoder to obtain the human motion prediction result of the target human body. The human motion prediction device further includes: The neural network encoder includes a kinematic encoder and a dynamic encoder. The kinematic encoder includes a first spatial Transformer and a first encoder-time Transformer. The dynamic encoder includes a second spatial Transformer and a second encoder-time Transformer. The neural network decoder includes a kinematic decoder and a dynamic decoder. The dynamic decoder includes a third spatial Transformer and a first decoder-time Transformer. The kinematic decoder includes a second decoder-time Transformer. The human motion prediction system also includes a quaternion transformation layer. Obtain the kinematic information of the target human body, and perform a preset simplified dynamics calculation on the kinematic information to obtain the dynamics information of the target human body; The kinematic information is combined with a preset spatial position embedding function and then input into the first spatial Transformer to obtain kinematic spatial features; the kinematic spatial features are flattened and combined with a preset encoder temporal position embedding function and then input into the first encoder temporal Transformer to obtain kinematic spatiotemporal features; the dynamic information is combined with a preset spatial position embedding function and then input into the second spatial Transformer to obtain dynamic spatial features; the dynamic spatial features are flattened and combined with a preset encoder temporal position embedding function and then input into the second encoder temporal Transformer to obtain dynamic spatiotemporal features; The last frame of encoded data corresponding to the kinematic information is copied a preset number of times and input into the third space Transformer to obtain the first query vector; the dynamic spatiotemporal features are used as the first key-value vector and input together with the first query vector into the first decoder time Transformer to obtain the second query vector; the kinematic spatiotemporal features are used as the second key-value vector and input together with the second query vector into the second decoder time Transformer to obtain the initial prediction result; the exponential mapping of the initial prediction result is converted into quaternions through the quaternion transformation layer, and training and optimization are performed in the quaternion space to obtain the human motion prediction result of the target human body.

5. A smart device, characterized in that, The intelligent device includes a processor, a storage unit, and a human motion prediction program stored on the storage unit that can be executed by the processor, wherein when the human motion prediction program is executed by the processor, it implements the steps of the human motion prediction method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a human motion prediction program, wherein when the human motion prediction program is executed by a processor, it implements the steps of the human motion prediction method as described in any one of claims 1 to 3.

Images