Human body action prediction method, system and device and storage medium

Abstract

The invention provides a human body motion prediction method, system and device and a storage medium, and belongs to the field of artificial intelligence, and the method comprises the steps: building mapping between a human body and a double-pendulum model, and obtaining a human body motion state and a corresponding motion parameter under the double-pendulum model; fitting a three-dimensional non-linear motion track of the double-pendulum model in a state space by using a cubic Bayesian spline according to the human body motion state at the historical moment and the corresponding motion parameters to obtain a non-linear kinetic equation; and predicting the motion state of the double pendulum at the next moment according to the nonlinear kinetic equation. Based on a conditional diffusion model, taking the human skeleton data and the double-pendulum motion state at the next moment as constraint conditions, and establishing a motion prediction model; and predicting a prediction result of the human body action of each part of the human body at the next moment through the motion prediction model. And the accuracy of human motion prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a human motion prediction method, system, device and storage medium. Background Art

[0002] Human motion prediction technology has important applications in multiple fields and is key to human-computer interaction in the intelligent era, including embodied intelligence, human-computer interaction, and digital human generation. Because human posture can become difficult to predict over time, the current work abstracts the actual need into a simplified task: predicting future human motion sequences based on observed motion sequences. Human motion prediction is a complex task involving multiple factors, including the combined effects of human physiology, motion patterns, and the external environment.

[0003] Traditional human motion prediction models extract features of motion sequences (such as speed, direction, and posture) and combine them with classifiers or regression models for prediction. However, this approach relies on strictly structured data and hand-designed features, and exhibits limitations when faced with high-dimensional, nonlinear human motion data. Insufficient data or low-quality data may cause the model to capture incorrect motion patterns, thereby affecting prediction accuracy. The continuous development of human motion prediction methods that incorporate physical laws can effectively improve the interpretability and rationality of prediction results and avoid generating motion sequences that do not conform to physical reality. By constructing a physical model to describe human motion and simulating the mechanical properties of the human body during motion, such as inertia, friction, and joint constraints, a strong physical guide is provided for motion prediction. The human body is a complex nonlinear system, but existing methods still face challenges when modeling the nonlinear motion of the human body in combination with physical laws. Summary of the Invention

[0004] In order to solve the problem of nonlinear motion in human motion prediction, the present invention provides a human motion prediction method, system, device and storage medium.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for predicting human motion comprises the following steps:

[0007] Obtain human skeleton data, establish a mapping relationship between the human body and the double pendulum model, and obtain the human body motion state and corresponding motion parameters under the double pendulum model;

[0008] Using cubic Bayesian splines to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space, a nonlinear dynamic equation is obtained; based on the human body motion state and corresponding motion parameters, the nonlinear dynamic equation is used to predict the double pendulum motion state of each part of the human body at the next moment;

[0009] A motion prediction model is constructed based on a conditional diffusion model; the human skeleton data and the double pendulum motion state at the next moment are input into the motion prediction model to obtain the human body movement at the next moment.

[0010] Preferably, the mapping relationship between the human body and the double pendulum model is specifically:

[0011] ψ=[x (1,t) ,y (1,t) ,x (2,t) ,y (2,t) ,θ (1,t) ,γ (1,t) ,θ (2,t) ,γ (2,t) ];

[0012] The double pendulum model consists of two massless rods with lengths l1 and l2, respectively. (1,t) represents the horizontal coordinate of the end point of rod l1 at time t on the xy plane; (1,t) represents the vertical coordinate of the end point of rod l1 at time t on the xy plane; (2,t) represents the horizontal coordinate of the end point of rod l2 at time t on the xy plane; (2,t) represents the ordinate of the end point of rod l2 at time t on the xy plane; θ (1,t) The rotation angle of rod l1 around the x-axis at time t; γ (1,t) represents the rotation angle of rod l1 around the y-axis at time t; θ (2,t) The rotation angle of rod l2 around the x-axis at time t; γ (2,t) represents the rotation angle of rod l2 around the y-axis at time t; the position of the hip joint is corresponded to the mass point at the starting end of the first rod of the double pendulum model, which is used as the origin, and the specific joints of the human limbs are aligned to the key points on the rod. The right knee joint is mapped to the end point of the first rod, and the ankle joint is mapped to the end point of the second rod.

[0013] Preferably, the method of fitting the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space using cubic Bayesian splines to obtain the nonlinear dynamic equation specifically includes the following steps:

[0014] According to the human joint control point C and the cubic Bayesian spline basis function H(t), the motion curve is obtained: x(t:C) = H(t)C, then the differential equation corresponding to the motion curve is:

[0015] According to the differential equation of the motion curve, the nonlinear dynamic equation is obtained as follows:

[0016]

[0017] Among them, φ(C) represents the function related to the human joint control point, selected from the function library φ; A is the coefficient matrix.

[0018] Preferably, the cubic Bayesian spline basis function is specifically:

[0019]

[0020] Among them, k S and k i are all segmentation points, i represents the number of cubic Bayesian spline basis functions, S represents the order; k represents any point in the computational domain.

[0021] Preferably, in the motion prediction model, the human body posture within t time steps is X 1:t ∈R J×3 , where J is the number of joints; the goal of human motion prediction is to map X to the human posture X′ of the future t′ frame. The goal of the motion prediction model is:

[0022] X′ t+1:t+n =I(X 1:t ,DDM(M(X 1:t )));

[0023] Where M represents the mapping from the human body posture space to the double pendulum state space, I represents the inverse mapping from the double pendulum state space back to the human body posture space, and DDM is a differentiable dynamics discovery model; n represents the time step;

[0024] The forward process of the motion prediction model introduces noise into the selected deterministic pose distribution, which is specifically expressed as:

[0025]

[0026] Among them, D~N(0,1) obeys the normal distribution from 0 to 1; G0 represents the deterministic posture distribution; represents the mean; represents variance;

[0027] In the reverse process of the motion prediction model, the denoising process guided by the constraint conditions is used to obtain the pose distribution G with high uncertainty. t Converge to the deterministic posture distribution G0; sample from the posture distribution G0 and generate the action prediction result of the whole body skeleton at the next moment at each time step.

[0028] The present invention also provides a human motion prediction system, which specifically includes:

[0029] The mapping module is used to obtain human skeleton data, establish a mapping relationship between the human body and the double pendulum model, and obtain the human body motion state and corresponding motion parameters under the double pendulum model.

[0030] The motion state module is used to use cubic Bayesian splines to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space to obtain a nonlinear dynamic equation; based on the human body motion state and corresponding motion parameters, the nonlinear dynamic equation is used to predict the double pendulum motion state of each part of the human body at the next moment.

[0031] The motion prediction module is used to build a motion prediction model based on the conditional diffusion model; the human skeleton data and the double pendulum motion state at the next moment are input into the motion prediction model to obtain the human body motion at the next moment.

[0032] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the disturbance suppression method of the quadrotor unmanned aerial vehicle system.

[0033] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is loaded by a processor, it can execute the steps described in the disturbance suppression method of the quadrotor unmanned aerial vehicle system.

[0034] The human body motion prediction method provided by the present invention has the following beneficial effects:

[0035] The present invention not only helps to capture the nonlinear motion of the human body by mapping the human body to the corresponding double pendulum model data, but also helps to fit flexible body parts such as arms and legs. By using cubic Bayesian splines to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space, a nonlinear dynamic equation is obtained, and the motion state and data of each part of the human body at the next moment are accurately predicted, reducing the error in the reconstruction between the human skeleton space and the double pendulum state space, and achieving accurate fitting of the nonlinear motion trajectory of the human body. A motion prediction model is constructed based on the conditional diffusion model, and the human skeleton data and the double pendulum motion state at the next moment are used as constraints to generate the prediction results of the human body's movements at the next moment. Effectively learn the relevant features of the human body's nonlinear motion and obtain continuous and high-quality human motion prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To more clearly illustrate the embodiments of the present invention and its design, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0037] Figure 1 The present invention is a flowchart of a method for predicting human motion.

[0038] Figure 2 : is a schematic diagram of a stage flow of a human motion prediction method according to an embodiment of the present invention, wherein: Figure 2 (a) is the overall process of human motion prediction. Figure 2 (b) is the processing flow of the dynamics discovery model. Figure 2 (c) is the processing flow of the motion prediction model. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solution of the present invention and to be able to implement it, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0040] Example

[0041] The present invention provides a method for predicting human motion. Figure 2 As shown in (a), it specifically includes the following steps:

[0042] S1. Map the human skeleton data to the corresponding double pendulum data.

[0043] Inspired by differentiable physical neural networks, a mapping between the human body and a double pendulum model is established. Compared with the simple single pendulum model, the double pendulum model is a typical nonlinear dynamic system that not only helps capture the nonlinear motion of the human body, but also helps fit flexible body parts such as arms and legs.

[0044] Specifically, the double pendulum model consists of two massless rods of lengths l1 and l2. The first rod has a mass at its starting point and end point, and the second rod is connected to the end point of the first rod and also has a mass at its end point. The state of the double pendulum model at time t can be expressed as:

[0045] ψ=[x (1,t) ,y (1,t) ,x (2,t) ,y (2,t) ,θ (1,t) ,γ (1,t) ,θ (2,t) ,γ (2,t) ];

[0046] Where (x, y) represents the coordinates of the end point of the rod in the xy plane; (θ, γ) represent the rotation angles of the rod around the x-axis and y-axis, respectively; (1, t) and (2, t) represent the states of rod 1 and rod 2 at time t, respectively.

[0047] In the mapping M, the hip joint is mapped to the mass at the beginning of the first rod, which serves as the origin. Subsequently, the human limbs are mapped into the double pendulum model, with specific joints aligned to key points on the rods. For the right leg, the knee joint is mapped to the end of the first rod, and the ankle joint is mapped to the end of the second rod. This simplified double pendulum model is used to study the nonlinear dynamics of human motion.

[0048] S2. Fit the double pendulum motion trajectory with a cubic Bayesian spline curve to obtain a nonlinear dynamic equation, and predict the double pendulum state data at the next moment based on the nonlinear dynamic equation.

[0049] The nonlinear dynamics discovery model is used to reveal the nonlinear dynamics of human motion. Given the transformed double pendulum model state, the transformed double pendulum model state parameters are used to reconstruct the three-dimensional nonlinear motion trajectory D of the transformed double pendulum model in the state space. m Due to the inherent differences between the human skeleton space and the double pendulum state space, as well as the reconstruction errors in the trajectory reconstruction process, the converted data exhibits a certain degree of sparsity and noise.

[0050] Physical information spline learning is used to predict the nonlinear dynamic equations from this sparse noisy double pendulum data. Specifically, cubic Bayesian splines are used to fit the reconstructed double pendulum motion trajectory. The cubic Bayesian spline is essentially a piecewise cubic polynomial function that is differentiable and continuous at the piecewise locations. The S-order of the i-th cubic Bayesian spline basis function H(k) is defined as:

[0051]

[0052] Among them, k s Indicates segmentation point.

[0053] Specify a set of control points C = (c1, c2, c3), select a suitable curve from multiple human motion trajectory curves through the control points to fit the motion trajectory, and define the motion curve x(t:C) by multiplying the control points C with the cubic Bayesian spline basis function H(t): x(t:C) = H(t)C.

[0054] The basis function H(t) is composed of multiple differentiable polynomials, so the differential equation of the motion curve x(t:C)=H(t)C can be derived.

[0055] For differentiable polynomials in the basis functions, the nonlinear dynamics are usually governed by a finite number of important terms, such as Figure 2 (b) is shown in Figure 2. Selecting important terms from the candidate function library φ, the nonlinear dynamic equation can be expressed as:

[0056]

[0057] Among them, φ(C)=φ(x(t:C)); A={λ1, λ2, λ3}∈R l×n is the coefficient matrix belonging to the constraint subset s.

[0058] For physical information spline learning, selecting an optimal set of control points will help to more accurately fit the nonlinear motion trajectory of the double pendulum. The goal of the nonlinear dynamics discovery model is to learn an optimal set of control points C and coefficient matrix A to best fit the motion data of the double pendulum while satisfying the nonlinear dynamics equation: Given motion data Among them, N m is the number of data points.

[0059] Reconstruct the double pendulum motion trajectory D using the historical motion state before time t m The nonlinear dynamic equations of the double pendulum model are learned using the cubic Bayesian spline function. The nonlinear dynamic equations are then used to predict the motion state of the double pendulum at time t+1, ψ(t+1).

[0060] S3. Based on the double pendulum motion state of each part of the human body at time step t+1 predicted by the nonlinear dynamics discovery model, the motion prediction model reconstructs the human skeleton from the double pendulum motion state space and predicts human movements.

[0061] Motion prediction stage, such as Figure 2 The goal of human motion prediction is to predict the future human posture based on the observed human posture sequence. Specifically, given the human posture X in t time steps, 1:t ∈R J×3 , where J is the number of joints. The goal of human motion prediction is to map X to the human pose X′ in the future t′ frame through the predictor F. The goal of the motion prediction model is:

[0062] X′ t+1:t+n =I(X 1:t ,DDM(M(X 1:t )));

[0063] Where M represents the mapping from the human posture space to the double pendulum state space, and I represents the posture reconstruction model, which is the inverse mapping from the double pendulum state space back to the human posture space. DDM is a differentiable dynamics discovery model designed to learn the nonlinear dynamics of the double pendulum.

[0064] like Figure 2 As shown in (c), the motion prediction model recovers the deterministic human skeleton from the highly uncertain double pendulum state space based on the conditional diffusion model. In the forward process, noise is added to the data distribution sampled from the human skeleton space to initialize an uncertain 3D posture distribution Gt In order to predict the target distribution from the noise distribution, supervision based on real data is essential. A series of 3D human motion data is used as an intermediate distribution to initialize the pose distribution. Subsequently, noise is gradually introduced into the selected deterministic pose distribution G0. The forward process of diffusion can be expressed as:

[0065]

[0066] Among them, D~N(0,I); G0 represents the deterministic posture distribution.

[0067] In the reverse process, the skeleton data X of the corresponding human body parts, the double pendulum motion state ψ(t+1) and the time step information are encoded into the conditional vector F DP The guided denoising process helps to extract the pose distribution G from the high uncertainty t Converges to a deterministic pose distribution G0. Then, sample from the pose distribution G0 to generate the next frame x of the full skeleton at each time step t+1 On this basis, the motion prediction model can be regarded as a U-net denoising network M that uses skeleton data, nonlinear double pendulum motion data, and time data as conditional constraints. The motion prediction model is trained by reconstructing the loss using the following formula.

[0068]

[0069] The human body movement at the next moment is predicted through the motion prediction model.

[0070] The present invention also provides a human motion prediction system, which specifically includes:

[0071] The mapping module is used to obtain human skeleton data, establish a mapping relationship between the human body and the double pendulum model, and obtain the human body motion state and corresponding motion parameters under the double pendulum model.

[0072] The motion state module is used to use cubic Bayesian splines to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space to obtain the nonlinear dynamic equation; based on the human body motion state and corresponding motion parameters, the nonlinear dynamic equation is used to predict the double pendulum motion state of each part of the human body at the next moment.

[0073] The motion prediction module is used to build a motion prediction model based on the conditional diffusion model; the human skeleton data and the double pendulum motion state at the next moment are input into the motion prediction model to obtain the human body motion at the next moment.

[0074] Each module in the human motion prediction system described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0075] The present invention also provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in an embodiment of a method for predicting human motion. The specific implementation method can be found in the method embodiment and will not be described in detail here.

[0076] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions, wherein a computer program is stored on the storage medium. For example, a memory containing instructions, the instructions can be executed by a processor of a computer device to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in an embodiment of a human motion prediction method can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0077] Those skilled in the art will appreciate that embodiments of the present invention may provide methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0081] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the present specification and examples have described the invention in detail, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the scope of protection of the patent for the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any person familiar with the art within the technical scope disclosed in the present invention falls within the scope of protection of the present invention.

Claims

1. A human motion prediction method, characterized in that: The following steps are involved: Obtain human skeleton data, establish a mapping relationship between the human body and the double pendulum model, and obtain the human body motion state and corresponding motion parameters under the double pendulum model; Using cubic Bayesian splines to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space, a nonlinear dynamic equation is obtained; based on the human body motion state and corresponding motion parameters, the nonlinear dynamic equation is used to predict the double pendulum motion state of each part of the human body at the next moment; A motion prediction model is constructed based on a conditional diffusion model; the human skeleton data and the double pendulum motion state at the next moment are input into the motion prediction model to obtain the human body movement at the next moment.

2. A human motion prediction method according to claim 1, characterized in that: The mapping relationship between the human body and the double pendulum model is specifically: The double pendulum model consists of two massless rods with lengths l1 and l2, respectively. (1,t) represents the horizontal coordinate of the end point of rod l1 at time t on the xy plane; (1,t) represents the vertical coordinate of the end point of rod l1 at time t on the xy plane; (2,t) represents the horizontal coordinate of the end point of rod l2 at time t on the xy plane; (2,t) represents the ordinate of the end point of rod l2 at time t on the xy plane; θ (1,t) The rotation angle of rod l1 around the x-axis at time t; γ (1,t) represents the rotation angle of rod l1 around the y-axis at time t; θ (2,t) The rotation angle of rod l2 around the x-axis at time t; γ (2,t) represents the rotation angle of rod l2 around the y-axis at time t; the position of the hip joint is corresponded to the mass point at the starting end of the first rod of the double pendulum model, which is used as the origin, and the specific joints of the human limbs are aligned to the key points on the rod. The right knee joint is mapped to the end point of the first rod, and the ankle joint is mapped to the end point of the second rod.

3. A human motion prediction method according to claim 1, characterized in that: The method of fitting the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space using cubic Bayesian splines to obtain the nonlinear dynamic equation specifically includes the following steps: According to the human joint control point C and the cubic Bayesian spline basis function H(t), the motion curve is obtained: x(t:C) = H(t)C, then the differential equation corresponding to the motion curve is: According to the differential equation of the motion curve, the nonlinear dynamic equation is obtained as follows: Among them, φ(C) represents the function related to the human joint control point, selected from the function library φ; A is the coefficient matrix.

4. A human motion prediction method according to claim 3, characterized in that: The cubic Bayesian spline basis function is specifically: Among them, k S and k i are all segmentation points, i represents the number of cubic Bayesian spline basis functions, S represents the order; k represents any point in the computational domain.

5. A human motion prediction method according to claim 1, characterized in that: In the motion prediction model, the human body posture within t time steps is X 1:t ∈R J×3 , where J is the number of joints; the goal of human motion prediction is to map X to the human posture X′ of the future t′ frame. The goal of the motion prediction model is: X′ t+1:t+n =I(X 1:t ,DDM(M(X 1:t ))); Where M represents the mapping from the human body posture space to the double pendulum state space, I represents the inverse mapping from the double pendulum state space back to the human body posture space, and DDM is a differentiable dynamics discovery model; n represents the time step; The forward process of the motion prediction model introduces noise into the selected deterministic pose distribution, which is specifically expressed as: Among them, D~N(0,1) obeys the normal distribution from 0 to 1; G0 represents the deterministic posture distribution; represents the mean; represents variance; In the reverse process of the motion prediction model, the denoising process guided by the constraint conditions is used to obtain the pose distribution G with high uncertainty. t Converge to the deterministic posture distribution G0; sample from the posture distribution G0 and generate the action prediction result of the whole body skeleton at the next moment at each time step.

6. A human motion prediction system, characterized in that: include: A mapping module is used to obtain human skeleton data, establish a mapping relationship between the human body and the double pendulum model, and obtain the human body motion state and corresponding motion parameters under the double pendulum model; A motion state module is used to fit the three-dimensional nonlinear motion trajectory of the double pendulum model in the state space using cubic Bayesian splines to obtain a nonlinear dynamic equation; based on the human body motion state and corresponding motion parameters, the nonlinear dynamic equation is used to predict the double pendulum motion state of each part of the human body at the next moment; The motion prediction module is used to build a motion prediction model based on the conditional diffusion model; the human skeleton data and the double pendulum motion state at the next moment are input into the motion prediction model to obtain the human body motion at the next moment.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is loaded into a processor, it can execute the steps of the method according to any one of claims 1 to 5.

Images