Human motion transition generation method, device and equipment and readable storage medium

Abstract

The application discloses a human motion transition generation method, device and equipment and a readable storage medium. The method comprises the following steps: obtaining an initial state, a real-time state, a target state and an expected state of a target human body; inputting the initial state, the real-time state, the target state and the expected state into a preset prediction model, and predicting a transition state between the initial state and the target state according to the preset prediction model to obtain a transition frame of the target human body; wherein the preset prediction model at least comprises a Gaussian process hidden variable model, and the Gaussian process hidden variable model is used to determine a posture change probability of a continuous action corresponding to the initial state, the real-time state and the target state; and performing local feature adjustment on the transition frame according to a preset adjustment strategy algorithm and the posture change probability to obtain a target frame. The application realizes the effect of improving the motion detail performance when generating a motion sequence.

Description

Technical Field

[0001] This application relates to the field of digital data technology, and in particular to a method, apparatus, device, and readable storage medium for generating human motion transitions. Background Technology

[0002] In computer animation, reusing existing motion clips offers advantages such as increased productivity and reduced costs. Motion transitions are typically created by searching data sets for the most suitable patterns for the context, and recent advancements in data-driven techniques can perfectly simulate temporal complexity.

[0003] However, when synthesizing periodic motions, the network may generate template sequences lacking biological vitality, and the generated transitions may lack detail, merely showing a stiff change in motion from the initial state to the target state. In particular, the network tends to directly use the accurate motion state learned during training as the transition state during motion switching, rather than creating a vivid and gradual transition under intentional control that incorporates the current motion context.

[0004] Therefore, when generating motion sequences, the generated motion sequences still suffer from insufficient representation of motion details compared to the actual motion postures. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, device and readable storage medium for generating human motion transitions, aiming to improve the expressiveness of motion details when generating motion sequences.

[0006] To achieve the above objectives, this application provides a method for generating human motion transitions, the method comprising the following steps:

[0007] Acquire the initial state, real-time state, target state, and expected state of the target human body;

[0008] The initial state, the real-time state, the target state, and the expected state are input into a preset prediction model, and the transition state between the initial state and the target state is predicted according to the preset prediction model to obtain the transition frame of the target human body.

[0009] The preset prediction model includes at least a Gaussian process latent variable model, which is used to determine the probability of posture change of continuous actions corresponding to the initial state, the real-time state and the target state.

[0010] Based on the preset adjustment strategy algorithm and the pose change probability, the transition frame is locally feature-adjusted to obtain the target frame.

[0011] For example, the preset prediction model includes a Gaussian process latent variable model and a temporal motion prediction model. The step of predicting the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body includes:

[0012] Based on the Gaussian process latent variable model, the real-time state is mapped to the latent variable manifold to obtain the latent state corresponding to the probability of change of each action in the real-time state;

[0013] The potential state, the initial state, the target state, and the expected state are input into the temporal motion prediction model, and the potential state, the initial state, the target state, and the expected state are feature-fused according to the temporal motion prediction model.

[0014] Based on the temporal motion prediction model and the fused features, the transition state between the initial state and the target state is predicted, and based on the transition state, the transition frame of the target human body is predicted.

[0015] For example, the step of feature fusion of the potential state, the initial state, the target state, and the expected state according to the temporal motion prediction model includes:

[0016] According to the temporal motion prediction model, context information is obtained by sampling from the potential state, the initial state, the target state, or the expected state in a temporal sequence.

[0017] The expected state is used as the target state when generating each frame of motion pose, and historical information of each frame of motion pose that has been generated is extracted.

[0018] According to the preset attention mechanism, the context information and the historical information are fused with the action posture corresponding to the expected state as the prediction direction.

[0019] For example, the step of adjusting the local features of the transition frame according to the preset adjustment strategy algorithm and the pose change probability to obtain the target frame includes:

[0020] According to a preset adjustment strategy algorithm, the adjacent frames of the transition frame are determined, and the context information of the action posture corresponding to the adjacent frame is determined; wherein, the context information of the action posture corresponding to the adjacent frame is sampled from the initial state, the target state, and the historical information;

[0021] Based on the context information, determine the change tendency of the transition frame in each dimension and in the latent variable manifold, and calculate the mean and covariance of the context information of the action poses corresponding to the adjacent frames;

[0022] Based on the mean, covariance, and pose change probability, the adjustment trend of the action pose corresponding to the transition frame is determined, and local feature adjustment is performed on the transition frame according to the adjustment trend to obtain the target frame.

[0023] For example, the step of mapping the real-time state to a latent variable manifold based on the Gaussian process latent variable model to obtain the latent state corresponding to the probability of change of each action in the real-time state includes:

[0024] Based on the Gaussian process latent variable model, the pose change features corresponding to each frame of the real-time state are extracted, and the pose change features are stacked into a first data matrix that includes the data dimension of each frame;

[0025] The real-time states are stacked into a second data matrix with a dimension lower than that of the pose change features;

[0026] According to the preset action dimensionality reduction mapping function, the first data matrix is ​​embedded into the second data matrix, and the embedded result is mapped to the latent variable manifold to obtain the latent state corresponding to the change probability of each action in the real-time state.

[0027] For example, before the step of embedding the first data matrix into the second data matrix according to a preset action dimensionality reduction mapping function, and mapping the embedded result to a latent variable manifold to obtain the latent state corresponding to the real-time state and the probability of each action change, the method includes:

[0028] Obtain the global dimension control parameters and arbitrary dimension control parameters associated with the potential dimension of the real-time state;

[0029] Based on the preset control formula, the global dimension control parameters, and the arbitrary dimension control parameters, the preset initial dimensionality reduction mapping function and the preset initial dimensionality increase mapping function are optimized to obtain the preset action dimensionality reduction mapping function and the latent state dimensionality increase mapping function; wherein, the latent state dimensionality increase mapping function is used to transform the latent state corresponding to the latent variable manifold into the actual action of the target human body.

[0030] For example, the step of inputting the initial state, the real-time state, the target state, and the expected state into a preset prediction model, and predicting the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body includes:

[0031] The initial state, the real-time state, and the target state are input into a preset prediction model, and action features corresponding to the real-time state are extracted based on the preset prediction model; wherein, the action features include at least action speed and action direction;

[0032] Based on the action speed, the action direction, and the preset prediction model, the transition state between the initial state and the target state is predicted, and based on the transition state and the potential state dimensionality-upgrading function, the transition frame of the target human body is predicted.

[0033] For example, before the step of obtaining the initial state, real-time state, target state, and expected state of the target human body, the following steps are included:

[0034] The movements of the target human body are captured to obtain a sequence of movement states of the target human body; wherein, each movement in the sequence of movement states is composed of the joint states of the preset root joint and other skeletal joints in the target human body;

[0035] The sequence of action states is processed into vector parameters to obtain the initial state, real-time state, and target state of the target human body;

[0036] Each element in the action state sequence is the pose data of the target human body. The pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of the other skeletal joints and the preset root joint.

[0037] For example, to achieve the above objectives, this application also provides a human motion transition generation device, the device comprising:

[0038] The acquisition module is used to acquire the initial state, real-time state, target state, and expected state of the target human body.

[0039] The prediction module is used to input the initial state, the real-time state, the target state, and the expected state into a preset prediction model, and predict the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body; wherein, the preset prediction model includes at least a Gaussian process latent variable model, and the Gaussian process latent variable model is used to determine the posture change probability of the continuous actions corresponding to the initial state, the real-time state, and the target state;

[0040] The adjustment module is used to perform local feature adjustment on the transition frame according to the preset adjustment strategy algorithm and the pose change probability to obtain the target frame.

[0041] For example, to achieve the above objectives, this application also provides a human motion transition generation device, the device comprising: a memory, a processor, and a human motion transition generation program stored in the memory and executable on the processor, the human motion transition generation program being configured to implement the steps of the human motion transition generation method as described above.

[0042] For example, to achieve the above objectives, this application also provides a computer-readable storage medium storing a human motion transition generation program, which, when executed by a processor, implements the steps of the human motion transition generation method as described above.

[0043] Compared to related technologies, where the generated motion sequences still lack sufficient representation of motion details compared to actual motion postures, this application addresses the issue by acquiring the initial state, real-time state, target state, and expected state of the target human body; inputting these states into a preset prediction model; and predicting the transition state between the initial state and the target state based on the preset prediction model to obtain a transition frame of the target human body. The preset prediction model includes at least a Gaussian process latent variable model (GLM), which determines the probability of posture changes in continuous actions corresponding to the initial state, real-time state, and target state. The transition frame is then adjusted according to a preset adjustment strategy algorithm and the posture change probability. Local feature adjustment yields the target frame. This involves using the initial state, real-time state, target state, and expected state as input to the prediction model. The model predicts the transition frame between the initial and target states. Based on the Gaussian process latent variable model within the prediction model, the probability of pose changes for the corresponding continuous actions in the initial, real-time, target, and expected states is determined. Furthermore, based on this pose change probability and a pre-defined adjustment strategy algorithm, the local features in the transition frame are adjusted. This avoids the situation where the network model generates a lifeless action sequence solely based on the action poses learned during training. Instead, it achieves the effect of adjusting the pose details of the generated transition frame based on the pose change probability of the target human's continuous actions, thus improving the expressiveness of motion details when generating motion sequences. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the first embodiment of the method for generating human motion transitions according to the applicant.

[0045] Figure 2 This is a flowchart illustrating the second embodiment of the method for generating human motion transitions according to the applicant.

[0046] Figure 3 This is a flowchart of a method and system for generating human motion transitions in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the process of generating transitional motion in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the network structure for learning temporal motion in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the spatial relationship of the implicit motion state in the embodiments of this application;

[0050] Figure 7 This is a schematic diagram illustrating the parameterization method of human motion capture data in the embodiments of this application;

[0051] Figure 8 This is a schematic diagram of human motion generated at three speeds in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram illustrating the generation of human motion from slow to fast forward in the embodiments of this application;

[0053] Figure 10 This application provides a schematic diagram of human motion generated from the same initial state to four target states in the embodiments of this application.

[0054] Figure 11 This application provides a schematic diagram of human motion generated from four initial states to the same target state in the embodiments of this application.

[0055] Figure 12 This is a schematic diagram illustrating the secondary motion changes generated according to the degree of randomness in the embodiments of this application;

[0056] Figure 13 This is a schematic diagram of the hardware operating environment involved in the embodiments of this application.

[0057] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0059] This application provides a method for generating human motion transitions, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for generating human motion transitions proposed by the applicant.

[0060] This application provides an embodiment of a method for generating human motion transitions. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order. For ease of description, the following omits the execution entity describing each step of the human motion transition generation method, which includes:

[0061] Step S110: Obtain the initial state, real-time state, target state, and expected state of the target human body;

[0062] For example, reusing existing motion clips in computer animation can improve production efficiency and reduce costs. This process can be achieved by setting an initial frame and a target frame for the animation. For instance, in a scene of target human movement, the initial frame can be the initial state of the target human, which is a state composed of the target human's action postures. This initial state can be a static state or an action posture at any moment in the action state. The target frame can be the target state of the target human, which can be a static state or an action posture at any moment in the action state. Based on the action postures or action states represented by the initial frame and the target frame of the animation, corresponding transition frames can be generated. In the form of transition frames, the initial frame, transition frames, and target frames are arranged in a continuous action process according to the corresponding time sequence (hereinafter referred to as timing), thereby completing the continuous motion clipping of the animation.

[0063] The initial state is the initial posture of the target human body in the process of the movement change, and the target state is the ending posture or target posture of the target human body in the process of the movement change. In other words, the overall trend of the movement change process is from the initial state to the target state.

[0064] Among them, the target human body is the target that needs to generate a continuous motion process. The target can also be a non-human body, such as an object that can generate motion relative to an absolutely stationary object. The initial state and target state corresponding to the target human body can include the action posture state that the human body can achieve, such as running, jumping, turning, walking, tilting the body to the side, tilting the body to the side, and so on. Each action corresponds to a specific action.

[0065] Furthermore, when generating continuous motion processes, relying solely on model-based motion deduction cannot simulate near-realistic motion changes. Therefore, in addition to obtaining the initial and target states, it is also necessary to acquire the corresponding real-time states. These real-time states can be obtained by pre-predicting the motion state of the target human body using a preset prediction model, or by real-time acquisition of the target human body's motion state. The real-time states include the motion sequence of the target human body in a continuous motion state, including at least the initial and target states, as well as the changes between the initial and target states. These real-time states can serve as reference data for the corresponding simulation process. They can be obtained by using the motion sequence pre-simulated by the preset prediction model as the real-time state, or by using the acquired real-time states of the target human body as parameter data. The acquired real-time states include capturing the corresponding motion posture, which is a fixed posture. For example, when the target human body walks, it needs to repeatedly lift and lower its legs, or it needs to perform small-amplitude, cyclical left and right arm swinging movements.

[0066] In addition, when generating a continuous motion process, relevant personnel need to set an expected state. This expected state is used to indicate the expected target state of the motion posture at each moment between the initial state and the target state. That is, the expected target state defines the desired motion posture effect for the relevant personnel. However, unlike the target state, the target state is only used to define the posture state at the last moment when the target human body ends the motion.

[0067] For example, a corresponding sequence of motions can be obtained from the initial state, the corresponding frames used to represent the motion transitions, and the target state. This sequence of motions is the final continuous motion animation containing the transition states that needs to be generated.

[0068] Step S120: Input the initial state, the real-time state, the target state and the expected state into the preset prediction model, and predict the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body;

[0069] The preset prediction model includes at least a Gaussian process latent variable model, which is used to determine the probability of posture change of continuous actions corresponding to the initial state, the real-time state and the target state.

[0070] The initial state, real-time state, target state, and expected state are input into a preset prediction model, which can then predict the transition state between the initial state and the target state, thereby obtaining the transition frame of the target human body.

[0071] In this process, when predicting the transition state between the initial state and the target state according to the preset prediction model, the predicted transition state still needs to be further processed to map the transition state to the corresponding actual transition action. Then, based on the transition action, an appropriate transition frame is generated. That is, the transition state represents the corresponding state change that needs to be represented in the transition, and the transition action is the specific action or posture that corresponds to the transition state.

[0072] The preset prediction model consists of two parts: a neural network model and a Gaussian process latent variable model. The Gaussian process latent variable model is mainly used to reduce the dimensionality of the original high-dimensional action posture or motion posture, thereby reducing the number of dimensions that need to be analyzed when analyzing action posture and removing redundant data in the original action posture motion analysis.

[0073] In other words, by using the Gaussian process latent variable model, motion posture is probabilistically embedded into a compact latent manifold. This manifold models the changes in motion state with motion uncertainty to construct the corresponding latent space. Essentially, the Gaussian process latent variable model constructs some inherent motion changes into a probabilistic, nonlinear, and nonparametric latent space. The trend probability of posture change corresponding to any action posture is obtained through the range of this manifold, and the magnitude of the trend probability of posture change corresponding to any action posture is represented by the depth of this manifold.

[0074] In summary, when the corresponding transition frame is output according to the preset prediction model, the action posture represented by the transition frame usually corresponds to the most likely action posture change. That is, it is equivalent to the influence of the posture change probability in the corresponding manifold obtained by combining the Gaussian process latent variable model. The initial state, real-time state, target state and expected state all correspond to different action postures. There are multiple possible action change sequences between the initial state and the target state. For example, taking the initial state as a stationary state and the target state as a running state as an example, when changing from a stationary state to a running state, there are at least two action change sequences. One is changing from a stationary state to a walking state and then switching from a walking state to a running state. The other is changing directly from a stationary state to a running state. When the above two types of action posture changes occur, there are different posture change probabilities. The transition frame will be generated in the best prediction way, that is, the case with the highest posture change probability will be selected as the corresponding action posture change of the transition frame.

[0075] Furthermore, since the probability of various pose changes is generated by predicting the expected state and the real-time state in combination with the initial state and the target state, the pose change amplitude between the initial state and the target state may be small. Therefore, when generating different action change sequences, the deviation between the various sequences representing different actions is small. That is, when outputting the transition frame, there are multiple optimal candidate action sequences. These candidate action sequences have similar pose change probabilities. During the prediction process, the candidate action sequences can be represented as the details of different action poses in the same type of behavior trend.

[0076] When outputting the transition frame, due to the processing logic of the neural network model, the action pose corresponding to the highest pose change probability will be output first as the action pose of the transition frame, while ignoring other action sequences with equally high pose change probabilities. Therefore, the currently output transition frame may not have the ability to express action details.

[0077] Step S130: According to the preset adjustment strategy algorithm and the pose change probability, perform local feature adjustment on the transition frame to obtain the target frame.

[0078] To further ensure that the generated transition frames can meet the detailed features of motion changes, the transition frames need to be adjusted accordingly. This adjustment process is based on the pose change probability of each state determined by the Gaussian process latent variable model, and according to the preset adjustment strategy algorithm, the motion pose represented by the transition frame is locally feature adjusted to address the uncertainty of the action represented by the pose change probability. In other words, after initially outputting the corresponding transition frames, the pose represented by each transition frame is refined and adjusted, breaking the original repetitive and single continuous motion sequence.

[0079] Meanwhile, the process of adjusting each transition frame based on the posture change probability mainly relies on adjusting the probability of all possible action postures corresponding to that transition frame. For example, taking any transition frame as an example, the action posture shown in the transition frame is a uniform walking state. Before the transition frame is generated, all possible action postures include uniform walking, accelerating walking, decelerating walking, or the body tilting slightly to the side during walking. Each of the above states corresponds to a posture change probability. At this time, the posture change probability corresponding to other states except uniform walking is adjusted, increasing the posture change probability corresponding to other states to output the benchmark with the highest posture change probability, thereby changing the output of the transition frame. That is, the action posture shown in the current transition frame should be the action posture corresponding to the maximum posture change probability after probability adjustment.

[0080] Therefore, based on the aforementioned preset adjustment strategy algorithm and pose change probability, one or more poses can be selected from the many possible poses, and their pose change probability can be amplified. This pose is then used as the pose to be represented in the new transition frame. Thus, based on the new pose, the local features of the transition frame are adjusted to obtain a new transition frame with corresponding action detail expression, rather than a fixed action sequence, i.e., the target frame.

[0081] In particular, when amplifying the probability of pose change of the action posture, a preset adjustment strategy algorithm is followed. The main purpose of this preset adjustment strategy algorithm is to use the parts of the action with a higher probability of pose change as candidate actions for adjusting the transition frame. The number of amplified pose change probabilities can be set according to actual needs.

[0082] In summary, traditional methods for generating continuous action sequences predict the sequence based on the initial and target states using a neural network model. However, the resulting action sequences often exhibit fixed and rigid variations compared to the actual action states. Therefore, in this embodiment, a preset prediction model is used, employing not only a neural network model but also a Gaussian process latent variable model to construct a framework simulating motion uncertainty. Within this framework, the original action sequence generation process is integrated with pose change probabilities to further refine and adjust transition frames, thereby enhancing the motion detail of the generated continuous action sequences.

[0083] Compared to related technologies, where the generated motion sequences still lack sufficient representation of motion details compared to actual motion postures, this application addresses the issue by acquiring the initial state, real-time state, target state, and expected state of the target human body; inputting these states into a preset prediction model; and predicting the transition state between the initial state and the target state based on the preset prediction model to obtain a transition frame of the target human body. The preset prediction model includes at least a Gaussian process latent variable model (GLM), which determines the probability of posture changes in continuous actions corresponding to the initial state, real-time state, and target state. The transition frame is then adjusted according to a preset adjustment strategy algorithm and the posture change probability. Local feature adjustment yields the target frame. This involves using the initial state, real-time state, target state, and expected state as input to the prediction model. The model predicts the transition frame between the initial and target states. Based on the Gaussian process latent variable model within the prediction model, the probability of pose changes for the corresponding continuous actions in the initial, real-time, target, and expected states is determined. Furthermore, based on this pose change probability and a pre-defined adjustment strategy algorithm, the local features in the transition frame are adjusted. This avoids the situation where the network model generates a lifeless action sequence solely based on the action poses learned during training. Instead, it achieves the effect of adjusting the pose details of the generated transition frame based on the pose change probability of the target human's continuous actions, thus improving the expressiveness of motion details when generating motion sequences.

[0084] For example, refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the applicant's method for generating human motion transitions. Based on the first embodiment of the applicant's method for generating human motion transitions described above, a second embodiment is proposed, wherein the method further includes:

[0085] Step S210: Based on the Gaussian process latent variable model, map the real-time state to the latent variable manifold to obtain the latent state corresponding to the change probability of each action in the real-time state;

[0086] In this embodiment, the preset prediction model includes a Gaussian process latent variable model and a time-series motion prediction model. The Gaussian process latent variable model is mainly used to construct the corresponding probabilistic, nonlinear, nonparametric latent space and embed inherent motion changes within it. For details, please refer to... Figure 3 (a)

[0087] For example, based on the Gaussian process latent variable model, the real-time state is mapped to the latent variable manifold. The spatial range occupied by the manifold and the corresponding color depth of the manifold are used as the probability of the action occurring at each moment corresponding to the real-time state, thereby obtaining the latent state.

[0088] In the Gaussian process latent variable model, the continuous action situation in real time is decomposed into a continuous action sequence. The posture and action changes of the action are analyzed moment by moment by using the order of occurrence of the action sequence.

[0089] Before using the Gaussian process latent variable model, the corresponding actions of the target human body are captured, and the captured action data is parameterized. Then, the latent variable manifold is established through the parameterized data, and the corresponding action data is implicitly expressed and analyzed through the latent variable manifold.

[0090] For example, firstly, we use a probabilistic latent variable model to represent motion uncertainty. Specifically, we first train the corresponding probabilistic latent variable model using samples from the relevant action dataset, thereby synthesizing a Gaussian process latent variable model. The motion dataset is Y = {y1, ..., y2}. N It contains N pose samples from all motion sequences.

[0091] Among them, y t That is, the change in each pose observed at time stamp t is represented by a latent state x with fewer dimensions. t .

[0092] The Gaussian process latent variable model constructs these inherent motion changes as probabilistic, nonlinear, and nonparametric latent space embeddings, the specific expression of which is:

[0093] y t =F GP (x t Y)

[0094] Among them, the latent variable x t Indicates y t Potential state changes.

[0095] Among them, F GP It contains unified information shared by Y, which is used to classify the latent variable x. t Reconstruct the frame data y t .

[0096] Among them, y is preserved through the Gaussian process latent variable model. tThe uncertainty is that, in the form of probability, the uncertainty of any action in Y is retained. This uncertainty refers to the uncertainty of how the action occurs. For example, the target human body may turn around in the next moment, or it may continue to walk forward in the next moment.

[0097] The inherent uncertainty is represented by each latent variable exhibiting a multivariate Gaussian distribution. Global changes are represented by the projections of the latent variables onto the manifold, while changes represented by the latent variables on the manifold correspond to changes in the expected state. This is achieved by constructing latent variables X = {x1, ..., x...}. N} and the potential mapping F from the observation dataset Y GP It was achieved.

[0098] In summary, the Gaussian process latent variable model constructs a latent space by taking samples of motion sequences and the uncertainty of each action in the samples, and maps the corresponding action sequences to the latent variable manifold according to the magnitude of their corresponding pose change probabilities.

[0099] Step S220: Input the potential state, the initial state, the target state and the expected state into the temporal motion prediction model, and perform feature fusion on the potential state, the initial state, the target state and the expected state according to the temporal motion prediction model;

[0100] By inputting the potential state, initial state, target state, and expected state into the temporal motion prediction model, the potential state, initial state, target state, and expected state can be fused using the temporal motion prediction model.

[0101] Specifically, a temporal motion prediction model is constructed based on recurrent neural units and attention modules. It learns temporal motion changes based on implicit motion state sequences (combined with the latent states of a Gaussian process latent variable model) and establishes contextual relationships across data segments. A corresponding neural network is then developed to learn the context of the state training samples. Since future motion is constrained by a deterministic context, the next motion state is defined as the current motion state x. t and control status For details, please refer to the function. Figure 3 (b)

[0102] The specific formula is as follows:

[0103] in, The control state indicates the expected motion target, expressed in first-order Markov form as the motion state x. t The model is based on the temporal relationships. The first-order assumption ensures contextual consistency while responding quickly to state changes.

[0104] For example, the step of feature fusion of the potential state, the initial state, the target state, and the expected state according to the temporal motion prediction model includes:

[0105] Step a: Based on the temporal motion prediction model, obtain context information by sampling from the potential state, the initial state, the target state, or the expected state in a temporal sequence.

[0106] Step b: Use the expected state as the target state when generating each frame of motion pose, and extract the historical information of each frame of motion pose that has been generated.

[0107] When fusing features of corresponding states according to a temporal motion prediction model, in order to ensure that the final transition frame output by the prediction model is as expected, an attention mechanism is set in the temporal motion prediction model. Through this attention mechanism, the corresponding potential state, initial state, target state, and expected state are fused, and the expected state is used as the target state when generating the action pose of each frame. Thus, as new transition frames are continuously generated, corresponding historical information is continuously produced. According to this attention mechanism, the relevant information of historical information and expected state can be further fused, which is equivalent to adjusting the weights of the data corresponding to different data fusion in the temporal motion prediction model. Through this attention mechanism, the weights are biased towards the expected state, that is, the action of the transition frame output by the temporal motion prediction model is more inclined to the action or pose corresponding to the expected state.

[0108] Among them, the corresponding context information can be sampled according to the time sequence from the initial state, target state and expected state. This context information is used as reference information when generating the next transition frame.

[0109] In this process, attention mechanisms are used to fuse contextual and historical information.

[0110] For example, refer to Figure 4 Based on the aforementioned latent variable manifold and context network, i.e. based on the preset prediction model, the system outputs a smooth motion transition that conforms to the natural state of the human body by inputting the initial state, real-time state, target state and expected state.

[0111] After the preset prediction model has been trained, the corresponding network circuit generates a single frame by receiving the new input state in each iteration. Assuming a transition occurs, the new target state is And historical information is X h ={..., x t Based on the specific timestamp, the currently input x tIt can be historical information of the current movement x t It can also be the target state c. t Another input By sampling the cascade sequence {X h C} is obtained. To output a natural and smooth transition, Based on the value of t, it can be divided into four stages. Specifically,

[0112]

[0113] or

[0114] or

[0115] or

[0116] That is, according to Figure 4 It can be seen that when fusing features of the initial state, target state, expected state and corresponding latent state, the latent state includes two parts: one is the relatively long action sequence shown in the real-time state, and the other is the action sequence corresponding to the initial state to the target state. The action sequence from the initial state to the target state is directly connected to the expected state. The initial state, target state and latent state are taken as one branch input, and the expected state is taken as another branch input. In the temporal motion prediction model, the two branch inputs are fused according to the attention mechanism, and the branch input corresponding to the expected state is taken as the main input direction of the final fused result.

[0117] Step c: Based on the preset attention mechanism, the context information and the historical information are fused with features in the direction of the action posture corresponding to the expected state.

[0118] The attention mechanism of the temporal motion prediction model is mainly implemented in the form of attention circuit modules.

[0119] For example, refer to Figure 5 It includes tensor embedding circuit units, timing loop circuit units, and attention units.

[0120] Specifically, motion context relationships are fitted by stacking circuit layers. It consists of an embedded circuit unit Θ and a history context circuit unit Φ. H A control context circuit unit Φ C It consists of an attention circuit unit Ω. The individual circuit units are described below:

[0121] The embedding unit Θ aims to align the data domains from the latent state X and the corresponding pose Y, thereby extracting fused features.

[0122] If historical state x is read in t The structure of the embedded unit Θ is as follows:

[0123]

[0124] Where MLP represents a multilayer perceptron circuit, and [;] represents vector cascading.

[0125] Θ is also used to control the context. For feature extraction, the parameters of the two circuit parts are shared. In this case, the specific expression is as follows:

[0126]

[0127] Furthermore, time consistency is learned by designing two context circuit units, satisfying the following two conditions: 1) only historical information is included and no future information is contained; 2) the expected state is used as the target state after the switch.

[0128] The first circuit unit is the history context unit Φ H It is created by a gated recurrent unit (GRU).

[0129] Φ H The product history information is read in at timestamp t, and then the potential state of the next frame is output:

[0130]

[0131] GRU1 is a two-layer unidirectional GRU.

[0132] It is the hidden state of GRU at timestamp t, and is updated in each iteration.

[0133] It should be noted that, Internal hidden state Control Context Unit Φ C Sure.

[0134] Another circuit unit is the unit control context circuit unit Φ. C It focuses on learning control sequences. Contextual clues. Φ C Structure and time context unit Φ H They are basically the same, but have a fixed iteration length M; at the same time, in order to extract rich contextual information from the forward and backward timestamps, Φ C Using a bidirectional cyclic GRU as the basic unit, and Φ H Only one-way.

[0135] Input control sequence Φ C The structure is

[0136] Wherein, GRU2 represents a two-layer bidirectional GRU. This indicates the hidden state of GRU2.

[0137] DYN internally merges contextual features Φ within the attention unit Ω. H (x t )and Output the predicted state that integrates historical data and control state.

[0138] For example, the attention module contains a query vector q, which serves as a unified basis for all components;

[0139] A key Z k As a reference group for q; and a value Z v The output data corresponds to the key.

[0140] Note the unit reads q and Z. k Based on the similarity between the two, Z is mixed. v Output q atn ,

[0141] The circuit unit Ω then further converts the original query q and the mixed output q involving keys and values. atn The components are then cascaded and input into an MLP (Multilayer Perceptron) for feature transformation. The specific transformation formula is as follows:

[0142] f atn (q, Z) K Z V ) = MLP4([q atn ;q]).

[0143] For example, select As key Z k Sum Z v Select historical context features Φ H (x t ) is used as query q.

[0144] Therefore, the expected motion state x is output at timestamp t+1. t+1 Represented as:

[0145]

[0146] In the final part of the internal circuit of the time-series motion prediction model, F GP By reading x t+1Restore the corresponding motion posture data y t+1 .

[0147] The above describes the internal circuit structure of the network in the temporal motion prediction model. The objective function for parameter optimization of the temporal motion prediction model is introduced below.

[0148] In terms of parameter optimization, the motion state x is supervised. t+1 and exercise posture y t+1 Optimize circuit parameters by matching the accuracy of the dataset.

[0149] The overall form of the loss function is

[0150] in, Represents the potential motion state x t+1 The loss of supervision, Indicates the posture of movement y t+1 The loss of oversight.

[0151] Hyperparameter λ modulation and The scale is set to 0.1 based on experience.

[0152] Loss during motion In this context, because the weights of each latent dimension contribute differently to the kernel measurement, a larger weight β... i The contribution of i∈[1,d] to supervision is greater.

[0153] For example, weighted accordingly Focus on key potential dimensions.

[0154]

[0155] Among them, DYN i This represents the i-th potential state dimension output by the temporal motion prediction model;

[0156] d is the number of potential dimensions.

[0157] T represents the length of the action segments in the training dataset.

[0158] x t+1 =(x t+1，1 , ..., x t+1，d ) is frame data y t+1 The corresponding latent variables.

[0159] It is the data center and x t Matching historical state sequence.

[0160] Postural loss Using motion state loss Similar forms:

[0161]

[0162] in, It is the data center and x t Matching historical state sequence.

[0163] Step S230: Based on the temporal motion prediction model and the fused features, predict the transition state between the initial state and the target state, and based on the transition state, predict the transition frame of the target human body.

[0164] Based on the temporal motion prediction model and the fused features, the corresponding transition frames of the target human body can be predicted. The number of transition frames depends on the number of frames of the corresponding action state change. For example, if the number of frames from the initial state to the target state is 15, then less than 15 transition frames need to be generated.

[0165] There are two scenarios when generating transition frames: First, a normal generation of 14 transition frames (taking 14 frames as an example; the number of transition frames can be other sizes, depending on the number of frames in the initial and target states; the number of frames in the initial and target states can be greater than 1). Second, if the number of transition frames generated has not yet reached 14, the generation of transition frames from the initial state to the target state is complete. In this case, to ensure the integrity of the transition frames generated by the preset prediction model, it is necessary to generate transition frames to complete the number of frames. For example, if the current generation of 5 transition frames completes the transition guidance, subsequent transition frames are generated. These transition frames, as supplementary frames, complete the number of frames from the initial state to the target state. The specific action state corresponding to the supplementary frame should coincide with the target state.

[0166] It should be noted that the initial state and the target state usually correspond to the corresponding actions within a single frame. At the same time, the initial state and the target state can also correspond to a sequence of multiple frames. For example, the target state is a sequence containing multiple action changes. The action changes do not need to be transitioned; only transition frames need to be generated between the initial state and the target state. Therefore, when generating supplementary frames, the supplementary frames can be overlapped with the target states of multiple frames to ensure the integrity of the generated continuous motion sequence.

[0167] The purpose of ensuring the integrity of the continuous motion sequence is as follows: When generating corresponding transition frames to complete the continuous motion sequence through a preset prediction model, the generated sequence can include multiple motion sequences in practical applications. For example, the input of the preset prediction model is multiple initial states and multiple target states. The corresponding initial states and target states form a group, forming at least two sets of motion sequences. This motion sequence can be from action A to action B, and then from action B to action C. That is, it is equivalent to needing to generate a continuous motion sequence from action A to action C. When generating the transition frame from action A to action B, if the generated transition frame arrives at action B in advance and no supplementary frame is generated, it will cause the transition from action B to action C to be discontinuous with the transition frame between action A and action B. Therefore, it is necessary to generate corresponding supplementary frames to ensure the integrity of the continuous motion sequence, thereby improving the effect of generating transition frames when using the preset prediction model in practical applications.

[0168] The specific actions corresponding to the transition frames have been explained above and will not be repeated here.

[0169] In summary, the preset prediction model includes a Gaussian process latent variable model and a temporal motion prediction model. When inputting relevant states to the prediction model, the corresponding states are first mapped according to the Gaussian process latent variable model and then mapped to the latent variable model to obtain the corresponding latent states. Based on the latent states and the temporal motion prediction model, the initial state, target state, latent state, and expected state are fused through an attention mechanism, and the corresponding transition frames are output according to the temporal motion prediction model.

[0170] After outputting the transition frame, the transition frame can be refined and adjusted using the latent variable manifold in the Gaussian process latent variable model to obtain the target frame.

[0171] For example, the step of adjusting the local features of the transition frame according to the preset adjustment strategy algorithm and the pose change probability to obtain the target frame includes:

[0172] Step d: According to the preset adjustment strategy algorithm, determine the adjacent frames of the transition frame, and determine the context information of the action posture corresponding to the adjacent frame; wherein, the context information of the action posture corresponding to the adjacent frame is sampled from the initial state, the target state and the historical information;

[0173] Step e: Based on the context information, determine the change tendency of the transition frame in each dimension and in the latent variable manifold, and calculate the mean and covariance of the context information of the action poses corresponding to the adjacent frames;

[0174] Step f: Based on the mean, covariance, and pose change probability, determine the adjustment trend of the action pose corresponding to the transition frame, and perform local feature adjustment on the transition frame according to the adjustment trend to obtain the target frame.

[0175] The sampling method for contextual information has been explained in the above description and will not be repeated here.

[0176] By combining the contextual information of the determined action state, the change tendency of the current transition frame in various dimensions and in the latent variable manifold can be determined, and the transition frame can be refined according to the change tendency.

[0177] Specifically, based on contextual information and change trends, the corresponding mean and covariance can be calculated. By combining the mean, covariance, and pose change probability, the action pose corresponding to the transition frame is adjusted. This adjustment process can determine the corresponding adjustment trend. For example, if the contextual information corresponds to an action pose, based on the action pose and the pose change probability of the current transition frame, other action trends, i.e., change trends, corresponding to the current action state of the current transition frame can be further determined. Based on these change trends, the difference between the action of the current transition frame and these change trends can be further determined, thereby determining the corresponding adjustment trend. Based on this adjustment trend, the local features of the transition frame can be adjusted to obtain the target frame.

[0178] For example, using mathematical principles, after generating the transition frame, a secondary adjustment to motion generation is needed to reduce repetitive patterns and generate a synthetic sequence with subtle variations, more like a biological sequence. Building upon the smooth transition, in this embodiment, motion with subtle variations can be generated based on probability sampling. These variations are categorized into uncertainty modeling of a Gaussian process and deterministic modeling of the network. Generative output can then be obtained by sampling from the uncertainty of the input latent state. Based on the single motion state variable x... t Multivariable Gaussian Its independence.

[0179] For example, x is defined in this embodiment. t Uncertainty Subject to the context of the surrounding area The impact, specifically expressed by the following formula:

[0180]

[0181] Since the product of Gaussians is still a Gaussian, The statistic can be expressed as a new Gaussian:

[0182] in:

[0183] In this embodiment, based on the Gaussian process latent variable model, the real-time state is mapped to a latent variable manifold to obtain the latent state corresponding to the probability of each action change. The latent state, the initial state, the target state, and the expected state are input into the temporal motion prediction model, and the latent state, the initial state, the target state, and the expected state are fused according to the temporal motion prediction model. Based on the temporal motion prediction model and the fused features, the transition state between the initial state and the target state is predicted, and the transition frame of the target human body is predicted based on the transition state. That is, the initial state, real-time state, target state, and expected state are processed accordingly by the Gaussian process latent variable model and the temporal motion prediction model in the prediction model, mapped to the latent variable manifold, and given relevant attributes of posture change probability. The input data is fused through the temporal motion prediction model. The input states are fused accordingly through an attention mechanism, so that the corresponding transition frame can be generated according to the corresponding fusion, ensuring the generated transition frame.

[0184] For example, based on the first and second embodiments of the applicant's human motion transition generation method described above, a third embodiment is proposed, wherein the method further includes:

[0185] Step g: Based on the Gaussian process latent variable model, extract the pose change features corresponding to each frame of the real-time state, and stack the pose change features into a first data matrix that includes the data dimension of each frame;

[0186] Step h: Stack the real-time states into a second data matrix with a dimension lower than that of the pose change features;

[0187] Step i: According to the preset action dimensionality reduction mapping function, embed the first data matrix into the second data matrix, and map the embedded result to the latent variable manifold to obtain the latent state corresponding to the change probability of each action in the real-time state.

[0188] When training the Gaussian process latent variable model, the original motion dataset is stacked into a data matrix by considering the pose of each frame. This data matrix is ​​the first data matrix;

[0189] Where N is the total number of frames;

[0190] D is the data dimension of each frame, which can be 94.

[0191] In this embodiment, the high-dimensional frame data Y is embedded into a compact low-dimensional latent state. This matrix is ​​the second data matrix, where d << D.

[0192] In this context, each frame data representing a change in action state is represented by a latent variable in probabilistic form, i.e., by a predefined initial dimensionality reduction mapping function. The form of the latent space constructed by the specific function can be found in [reference needed]. Figure 6 Its specific expression is as follows:

[0193]

[0194] Where tr(·) represents the trace of the matrix;

[0195] The kernel matrix K is the covariance of the latent variable X, and it is also a measure of the change of the latent state in the latent space; X is the latent state.

[0196] Here, the potential state represents any possible state in the potential space. For example, between the initial state and the target state, it is possible for walking to turn into a brisk walking state, or walking to turn into a running state, or walking to turn into a turning motion, etc. Each possible state is represented by the probability of each possible state as a transition state.

[0197] When generating transition frames based on the latent state and other states, the confidence level of the latent state is considered. This confidence level can be obtained by statistically analyzing the pose change probability corresponding to the latent state. Typically, this confidence level can be the mean of the pose change probability.

[0198] For example, before the step of embedding the first data matrix into the second data matrix according to a preset action dimensionality reduction mapping function, and mapping the embedded result to a latent variable manifold to obtain the latent state corresponding to the real-time state and the probability of each action change, the method includes:

[0199] Step j: Obtain the global dimension control parameters and arbitrary dimension control parameters associated with the potential dimension of the real-time state;

[0200] Step k: Based on the preset control formula, the global dimension control parameters, and the arbitrary dimension control parameters, optimize the preset initial dimensionality reduction mapping function and the preset initial dimensionality increase mapping function to obtain the preset action dimensionality reduction mapping function and the latent state dimensionality increase mapping function; wherein, the latent state dimensionality increase mapping function is used to transform the latent state corresponding to the latent variable manifold into the actual action of the target human body.

[0201] In Gaussian process latent variable models, the squared exponential kernel follows the concept of automatic correlation determination to improve the model's scalability.

[0202] The frame data (the transition frame obtained from each frame determination) is represented by the probabilistic form P(Y|X, α) of the latent state, where a common choice for K is the radial basis function.

[0203] For example, according to the radial basis kernel, each element k(x, x′) of K is calculated by any latent variables X1, X2 and hyperparameter α, and the specific calculation formula (preset control formula) is as follows:

[0204]

[0205] For example, the radial basis kernel can follow the Automatic Correlation Determination (ARD) algorithm to improve the scalability of the model.

[0206] The ARD parameters α = (γ, β) learn the length scale associated with the dimension of each latent state to emphasize more discriminative data dimensions. γ scales the kernel globally, i.e., γ is a global dimension control parameter, while β = (β1, ..., β) d Independently control the corresponding dimension x1 of any latent variable x1 = (x 1，1 , ..., x 1，d ), where β is an arbitrary dimension control parameter. Using the ARD algorithm, more dimensions are activated to represent complex motion structures, or some dimensions are activated to handle simple structures.

[0207] In order to avoid loss of generality, a separate prior distribution can be applied to the above parameters.

[0208] Therefore, the loss function for optimizing the Gaussian process latent variable model by minimizing the negative log-posterior is:

[0209]

[0210] in, It is proportional to the corresponding negative log-likelihood.

[0211] We use maximum a posteriori probability (MAP) to optimize the latent input X and kernel parameter α.

[0212] Based on the optimization results, input any latent variable x. ＊ The preset prediction model can output the corresponding motion frame data y ＊ The output is in the form of a closed loop. Gaussian distribution:

[0213] μ(x ＊ ) = F GP (x ＊ Y) = K x K -1 Y,

[0214]

[0215] Where K x By Y and x ＊ Substituting into the kernel function yields the result.

[0216] The mean μ represents the value from x ＊ Average estimation of moving frames;

[0217] The variance ∑ represents the confidence level. The lower the variance, the higher the confidence level of the motion frame reconstruction, and vice versa.

[0218] In summary, the aforementioned global and arbitrary dimension control parameters can be used to control the specific computational effect of the mapping function. By adjusting global variables and refining precise variables, the mapping accuracy of the mapping function between action state, action posture, and posture change probability can be improved. The mapping function includes a preset action dimensionality reduction mapping function and a latent state dimensionality increase mapping function. The preset action dimensionality reduction mapping function is mainly used to map the action corresponding to the actual action state into a motion latent state represented by posture change probability in the latent variable manifold. The latent state dimensionality increase mapping function is mainly used to map the motion latent state represented by posture change probability in the latent variable manifold to a specific motion posture or motion action. That is, through the mapping function, a mapping relationship between motion action and latent state is established, and accurate conversion between the two is achieved, thereby improving the accuracy and reasonableness of the preset prediction model in generating transition frames based on the aforementioned data.

[0219] In this embodiment, when processing the corresponding real-time state according to the Gaussian process latent variable model, the main purpose is to reduce the dimensionality of the real-time state and its corresponding action. That is, the action or pose is taken as the dataset Y and embedded into the low-dimensional latent state. In this way, the corresponding real-time state is expressed in the form of a probability function. A preset control formula is designed to optimize the preset initial dimensionality reduction mapping function expression. Through global dimension control parameters and arbitrary dimension control parameters, the parameters are controlled from the comprehensive global perspective and the parameters corresponding to different dimensions, respectively, to improve the dimensionality reduction effect of the Gaussian process latent variable model.

[0220] For example, based on the first, second, and third embodiments of the applicant's human motion transition generation method described above, a fourth embodiment is proposed, wherein the method further includes:

[0221] Step 1: Capture the movements of the target human body to obtain a sequence of movement states of the target human body; wherein, each movement in the sequence of movement states is composed of the joint states of the preset root joint and other skeletal joints in the target human body;

[0222] Step n: Perform vector parameterization on the action state sequence to obtain the initial state, real-time state, and target state of the target human body;

[0223] Each element in the action state sequence is the pose data of the target human body. The pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of the other skeletal joints and the preset root joint.

[0224] Before obtaining the corresponding initial state, real-time state, target state, and expected state, it is necessary to first obtain the corresponding initial state and target state. At the same time, different expression methods should be set according to different states, and the expression methods of the above-mentioned states should be determined based on the central idea of ​​reducing the dimension of action states.

[0225] The process involves capturing the movements of the target human body and obtaining a sequence of the target human body's movement states based on the captured movement-related data. From this sequence of movement states, the initial state, target state, and real-time state are extracted accordingly.

[0226] Each action in the action state sequence consists of the joint states of the preset root joint and other skeletal joints in the target human body. Each element in the action state sequence is the pose data of the target human body. This pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of other skeletal joints and the preset root joint.

[0227] In this embodiment, the preset root joint is a reference joint point specifically set for the target human body. The preset root joint has six dimensions, while other skeletal joints have three dimensions.

[0228] Specifically, the preset root joint can be the waist position at the center of the human body. This waist position has a corresponding movement dimension in three-dimensional space, as well as angular velocity and movement speed. The preset root joint is used as a reference point, and the relative position coordinates of other skeletal joints with the preset root joint are generated based on the reference point.

[0229] Reference Figure 7 This includes the interrelationship between human motion, BVH motion capture data (BVH is the file extension of the file format generated after human motion is captured by devices such as BioVision), and motion expression parameters of the adaptation algorithm.

[0230] Motion capture data is typically saved as a BVH file, recording the position and rotation of a preset root joint, as well as the rotation of other skeletal joints in the target human body. However, this industry-standard data structure does not meet the requirements for refined data-driven models. Therefore, in this embodiment, the BVH data that would normally be used is combined with the relative position coordinates of other skeletal joints to represent the topology of the human skeleton.

[0231] For example, the motion dataset used by the preset prediction model is represented as follows: Each original sample It consists of 31 joints. The first six dimensions represent the preset root joint position and rotation, while the remaining dimensions represent the rotational movements of other skeletal joints.

[0232] Since Y contains information from global coordinates, it needs to be parsed into local character coordinates for model training.

[0233] in, Vectorized into {y t} Each row of Y represents the t-th pose data.

[0234] Among them, each pose y t Including root motion velocity Relative position to other joints

[0235] The root motion is defined as the displacement velocity along the xyz axes. and angular velocity on the y-axis

[0236] The other 30 joints are represented from the root position. The initial corresponding 3D relative position coordinates. Therefore, each pose is represented as... ([;] indicates vector concatenation).

[0237] For example, the original time data is obtained from joint positions using inverse kinematics (IK). In this application, the original motion posture is recovered by following the IK method. Specifically, the preset root joint IK integrates historical root movements. For other joints, IK refers to their relative position to the root.

[0238] In this embodiment, the movements of the target human body are captured to obtain a sequence of movement states of the target human body. Each movement in the sequence consists of the joint states of a preset root joint and other skeletal joints in the target human body. The sequence of movement states is then processed into vector parameterization to obtain the initial state, real-time state, and target state of the target human body. Each element in the sequence of movement states is the pose data of the target human body. The pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of the other skeletal joints to the preset root joint. That is, by setting a corresponding preset root joint and using it as a reference point, the relative position coordinates between the other skeletal joints and the preset root joint are determined. This transforms the original movement data into state movements composed of the joints of the target human body, thereby reducing the dimensionality of the state data and reducing the amount of redundant data when processing the movement data.

[0239] For example, the step of inputting the initial state, the real-time state, the target state, and the expected state into a preset prediction model, and predicting the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body includes:

[0240] Step o: Input the initial state, the real-time state, and the target state into a preset prediction model, and extract the action features corresponding to the real-time state based on the preset prediction model; wherein, the action features include at least action speed and action direction;

[0241] Step p: Based on the action speed, the action direction, and the preset prediction model, predict the transition state between the initial state and the target state, and based on the transition state and the potential state dimensionality-upgrading function, predict the transition frame of the target human body.

[0242] The initial state, real-time state, and target state are used as inputs to a preset prediction model. Based on the preset prediction model, the action features corresponding to the real-time state are extracted. These action features include at least action speed and action direction. Based on the action speed and action direction, the transition state between the initial state and the target state is predicted to obtain the transition frame of the target human body.

[0243] That is, to further ensure the authenticity of the transition frames predicted at present, control conditions can be added accordingly. For example, the direction and speed of movement in real time can be used as reference data for prediction, and the probability of posture change under the influence of the direction and speed of movement can be calculated comprehensively, thereby increasing the breadth of the calculated probability and the accuracy of the prediction of the movement.

[0244] In addition, action characteristics also include action cycles (e.g., the repetitive cycle of walking movements, including the cycle of repetitive movements of the legs, arms, etc. during walking) and action tendencies (e.g., the presence of a target human body will trigger corresponding preparatory movements when the body turns to the side or undergoes a large change in body position, requiring the legs to build up strength or turning the neck first), which are used to describe changes in the motion state of the target human body.

[0245] Reference Figure 8 The figure shows the synthesized results of three initial sequences at different speeds: slow (first row), medium (second row), and fast (third row). The latent variables for synthesizing the motion sequences are as follows: Figure 8 As shown in (a), different shades represent different confidence scores for the Gaussian process reconstruction of a reasonable walking posture, with darker shades indicating high confidence and lighter shades indicating low confidence. All latent variables are located in regions with high confidence, implying a reasonable quality of posture reconstruction. Simultaneously, each group of latent variables clusters in a specific region, indicating that postures at different walking speeds are clearly separated and have no overlap. The circles formed by the latent paths explain the periodicity of walking. Results of the motion visualization reconstruction of the avatar. Figure 8 (c) shows the reconstruction quality in the full view. After constructing a 1.5-second composite at 60Hz, the visualization frames were sampled at a frame rate of 6Hz. The difference in motion speed, considering variations between adjacent frames, still exists, demonstrating the stability of the composite. Figure 8 The curve in (b) illustrates how the height of the right knee changes over time. The motion pattern of the individual joint repeats periodically, demonstrating synthetic stability in the partial view.

[0246] Reference Figure 9 The gait variation is achieved by connecting clips with different velocity states. The transitions between different velocities in the figure show that the initial state (left incarnation) is relaxed and slow, while the target state (right incarnation) is tense and fast. The transition process (skeleton) is smooth and natural, without drastic changes. Even without ground truth supervision, our model can still smoothly transition from one state to another. This is because it contributes to latent embeddings and contextual state control. On the one hand, F... GP The latent embeddings distinguish between minute noises from motion uncertainty and significant changes in motion states, which helps to find intrinsic differences and gradually blend them over time. On the other hand, the attention mechanism of the currently proposed network adaptively selects the control state that contributes most to the real environment, thereby maintaining temporal consistency and preventing abrupt changes in joint paths.

[0247] In this embodiment, the initial state, the real-time state and the target state are input into a preset prediction model, and action features of the action corresponding to the real-time state are extracted according to the preset prediction model, where the action features at least include an action speed and an action direction; according to the action speed, the action direction and the preset prediction model, a transition state between the initial state and the target state is predicted, and a transition frame of the target human body is predicted and obtained according to the transition state and a potential state dimension-increasing function, that is, when the transition frame and the target frame are generated through the preset prediction model, the probability value of the action corresponding to each state determined by a Gaussian process latent variable model can be used to control the action speed and the action direction of the actions corresponding to the generated transition frame and target frame, so as to improve the authenticity of the continuous action sequence generated by the preset prediction model.

[0248] As an example, based on the first, second, third and fourth embodiments of the human motion transition generation method of the present application described above, a fifth embodiment is proposed, which includes:

[0249] In this embodiment, the experimental data of the preset prediction model is taken as a reference to perform a corresponding analysis on the effect achieved by the preset prediction model.

[0250] Reference Figure 10 , which includes a schematic diagram of initial states, target states and centroid trajectories of four motion sequences in the same scene, and a schematic diagram of the respective frame sequences of the four motions.

[0251] This test generates transitions from one source to multiple targets. The immediate response of the model is evaluated by setting different target states.

[0252] As an example, the direction change is connected to the same initial state through specific parameter settings. The figure illustrates direction change control starting from the same timestamp. Unlike walking straight forward, the turning direction is more complex in lower body motion. It can be based on Figure 10 observed that the preset prediction model gives the same fast response to turning back, for example, Figure 10 left (b) of, Figure 10 straight line (c) of, and correct Figure 10 (d) of.

[0253] Wherein, the expected state contains a specific context with an exact timestamp, which pushes the transition of each step to the maximum instead of considering ambiguous and weak control signals. At the same time, the recurrent context unit ensures that context continuity is naturally performed. Due to the gradual contribution of state connection, joint angles transition smoothly during the synthesis sequence, rather than undergoing sharp and sudden transformations by connecting motion clips.

[0254] It should be noted that no motion context was initially recorded in the training data used during the initial training of the preset prediction model. That is, in the unknown transition context, the biases that occur are adjusted by the stochasticity of the Gaussian process latent variable model. The current action transition only occurs within the manifold geometry, therefore the context can reference nearby latent states, indicating similarity to the expected pose.

[0255] refer to Figure 11 It includes the initial state, target state and centroid trajectory diagrams of four motion sequences in the same scene, and the frame sequence diagrams of each of the four motion sequences.

[0256] This test generates transitions from multiple sources to a single target. The model's accuracy is evaluated by setting different initial states while simultaneously targeting the same target state. In fact, connecting various states to the same target should result in similar action transitions in the final output. That is, regardless of changes in prior context information, the final output state of the action sequence's context must remain constant.

[0257] The experimental results, starting with random walking in a specific direction, are shown in the figure. The example of the target human walking backwards is used to illustrate the movement pattern, which is from left to right. Figure 11 Starting from (b), Figure 11 Starting from (c) in front of, Figure 11 Starting from the right (d), all characters turn left. The composition ends with the exact same cut because all characters turn in the same direction and walk at the same speed.

[0258] refer to Figure 12 This includes schematic diagrams of motion changes with low and high randomness in a rotating sequence, and schematic diagrams of motion changes with low and high randomness in a continuous changing sequence.

[0259] For similar expected states, synthesis should be performed with small but precise variations. Preset prediction models can apply detailed variations to edit secondary motions while maintaining general motion patterns.

[0260] In this embodiment, a preset prediction model generates a similar state sequence from the uncertainty of the input state. For example... Figure 12 As shown in (a1), the corresponding target human body's repetitive motion states are set up in a stable angular velocity loop. Since this state sequence is copied from the same clip, the walking path overlaps in each loop. When generating latent states from the original states, the character's arm and leg movements in the loop have more variation, which is different for each walking loop; see reference for details. Figure 12 (a2). Overall, the synthetic motion remains a smooth and natural cycle. And... Figure 12(b1) is another context sequence in which a longer motion is generated and a consistent rotation is maintained. This motion sequence is regenerated through uncertainty sampling, specifically as follows: Figure 12 As shown in (b2).

[0261] In this scenario, the target human body maintains a corresponding turning trend at each turn, while simultaneously altering its walking path and detailed body movements. The corresponding uncertainty generation strategy is an option for reducing template-based synthetic results.

[0262] In this embodiment, it is difficult to generate controlled synthetic motion because the network output is determined by its input. To address this, a pre-defined predictive model is used to provide a method for diverse motion transitions by sampling from the uncertainty of the input state. Simultaneously, the distribution of the motion context is defined based on a Gaussian process latent variable model, and the uncertainty of this motion context guarantees the generation quality. Furthermore, the pre-defined predictive model distinguishes high-quality motion with high confidence, which is implicitly reflected in the statistics. Therefore, the motion sequences generated based on the uncertainty of the pose change probability ensure higher detail quality.

[0263] In addition, this application also provides a human motion transition generation device, which includes:

[0264] The acquisition module is used to acquire the initial state, real-time state, target state, and expected state of the target human body.

[0265] The prediction module is used to input the initial state, the real-time state, the target state, and the expected state into a preset prediction model, and predict the transition state between the initial state and the target state according to the preset prediction model to obtain the transition frame of the target human body; wherein, the preset prediction model includes at least a Gaussian process latent variable model, and the Gaussian process latent variable model is used to determine the posture change probability of the continuous actions corresponding to the initial state, the real-time state, and the target state;

[0266] The adjustment module is used to perform local feature adjustment on the transition frame according to the preset adjustment strategy algorithm and the pose change probability to obtain the target frame.

[0267] For example, the prediction module includes:

[0268] The mapping submodule is used to map the real-time state to the latent variable manifold according to the Gaussian process latent variable model, so as to obtain the potential state corresponding to the change probability of each action in the real-time state.

[0269] The fusion submodule is used to input the potential state, the initial state, the target state and the expected state into the temporal motion prediction model, and to perform feature fusion on the potential state, the initial state, the target state and the expected state according to the temporal motion prediction model;

[0270] The first prediction submodule is used to predict the transition state between the initial state and the target state based on the temporal motion prediction model and the fused features, and to predict the transition frame of the target human body based on the transition state.

[0271] For example, the fusion submodule includes:

[0272] The sampling unit is used to obtain context information by sampling from the potential state, the initial state, the target state, or the expected state according to the temporal motion prediction model.

[0273] The first extraction unit is used to take the expected state as the target state when generating each frame of motion pose, and to extract the historical information of each frame of motion pose that has been generated.

[0274] The fusion unit is used to fuse the context information and the historical information according to a preset attention mechanism, with the action posture corresponding to the expected state as the prediction direction.

[0275] For example, the adjustment module includes:

[0276] The first determining submodule is used to determine the adjacent frames of the transition frame according to a preset adjustment strategy algorithm, and to determine the context information of the action posture corresponding to the adjacent frame; wherein, the context information of the action posture corresponding to the adjacent frame is sampled from the initial state, the target state and the historical information;

[0277] The second determining submodule is used to determine the change tendency of the transition frame in each dimension and in the latent variable manifold based on the context information, and to calculate the mean and covariance of the context information of the action poses corresponding to the adjacent frames;

[0278] The third determining submodule is used to determine the adjustment trend of the action posture corresponding to the transition frame based on the mean, covariance and the posture change probability, and to perform local feature adjustment on the transition frame according to the adjustment trend to obtain the target frame.

[0279] For example, the mapping submodule includes:

[0280] The second extraction unit is used to extract the pose change features corresponding to each frame of the real-time state according to the Gaussian process latent variable model, and stack the pose change features into a first data matrix that includes the data dimension of each frame.

[0281] A transformation unit is used to stack the real-time states into a second data matrix with a dimension lower than that of the pose change features;

[0282] The mapping unit is used to embed the first data matrix into the second data matrix according to the preset action dimensionality reduction mapping function, and map the embedded result to the latent variable manifold to obtain the latent state corresponding to the change probability of each action in the real-time state.

[0283] For example, the mapping unit includes:

[0284] The acquisition subunit is used to acquire the global dimension control parameters and arbitrary dimension control parameters associated with the potential dimension of the real-time state;

[0285] The optimization subunit is used to optimize the preset initial dimensionality reduction mapping function and the preset initial dimensionality increase mapping function according to the preset control formula, the global dimension control parameters and the arbitrary dimension control parameters, so as to obtain the preset action dimensionality reduction mapping function.

[0286] For example, the device includes:

[0287] The capture module is used to capture the movements of the target human body and obtain a sequence of movement states of the target human body; wherein, each movement in the sequence of movement states is composed of the joint states of the preset root joint and other skeletal joints in the target human body;

[0288] The processing module is used to perform vector parameterization processing on the action state sequence to obtain the initial state, real-time state and target state of the target human body; wherein, each element in the action state sequence is the pose data of the target human body, and the pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of the other skeletal joints and the preset root joint.

[0289] For example, the prediction module further includes:

[0290] An extraction submodule is used to input the initial state, the real-time state, and the target state into a preset prediction model, and extract the action features corresponding to the real-time state based on the preset prediction model; wherein, the action features include at least action speed and action direction;

[0291] The second prediction submodule is used to predict the transition state between the initial state and the target state based on the action speed, the action direction and the preset prediction model, so as to obtain the transition frame of the target human body.

[0292] The specific implementation method of the human motion transition generation device of this applicant is basically the same as the embodiments of the above-mentioned human motion transition generation method, and will not be repeated here.

[0293] In addition, this application also provides a human motion transition generation device. For example... Figure 13 As shown, Figure 13 This is a schematic diagram of the hardware operating environment involved in the embodiments of this application.

[0294] For example, Figure 13 This is a schematic diagram of the hardware operating environment of the device that generates transitions for human movement.

[0295] like Figure 13 As shown, the human motion transition generation device may include a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The memory 1303 is used to store computer programs. When the processor 1301 executes the program stored in the memory 1303, it implements the steps of the human motion transition generation method.

[0296] The communication bus 1304 mentioned in the aforementioned human motion transition generation device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1304 can be divided into an address bus, a data bus, and a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0297] The communication interface 1302 is used for communication between the aforementioned human motion transition generation device and other devices.

[0298] The memory 1303 may include random access memory (RMD) or non-volatile memory (NM), such as at least one disk storage device. Optionally, the memory 1303 may also be at least one storage device located remotely from the aforementioned processor 1301.

[0299] The processor 1301 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0300] The specific implementation method of the human motion transition generation device of this applicant is basically the same as the embodiments of the above-mentioned human motion transition generation method, and will not be repeated here.

[0301] Furthermore, this application also proposes a computer-readable storage medium storing a human motion transition generation program, which, when executed by a processor, implements the steps of the human motion transition generation method described above.

[0302] The specific implementation of the computer-readable storage medium in this application is basically the same as the embodiments of the above-described human motion transition generation method, and will not be repeated here.

[0303] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0304] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0305] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0306] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for generating human motion transitions, characterized in that, The method for generating human motion transitions includes the following steps: Acquire the initial state, real-time state, target state, and expected state of the target human body; The initial state, the real-time state, the target state, and the expected state are input into a preset prediction model, and the transition state between the initial state and the target state is predicted according to the preset prediction model to obtain the transition frame of the target human body. The preset prediction model includes at least a Gaussian process latent variable model, which is used to determine the probability of posture change of continuous actions corresponding to the initial state, the real-time state and the target state. Based on the preset adjustment strategy algorithm and the pose change probability, the transition frame is locally feature adjusted to obtain the target frame; The preset prediction model includes a Gaussian process latent variable model and a temporal motion prediction model. The step of predicting the transition state between the initial state and the target state based on the preset prediction model to obtain the transition frame of the target human body includes: Based on the Gaussian process latent variable model, the real-time state is mapped to the latent variable manifold to obtain the latent state corresponding to the probability of change of each action in the real-time state; The potential state, the initial state, the target state, and the expected state are input into the temporal motion prediction model, and the potential state, the initial state, the target state, and the expected state are fused based on the temporal motion prediction model. Based on the temporal motion prediction model and the fused features, the transition state between the initial state and the target state is predicted, and the transition frame of the target human body is predicted based on the transition state. The step of adjusting the local features of the transition frame according to the preset adjustment strategy algorithm and the pose change probability to obtain the target frame includes: According to the preset adjustment strategy algorithm, the adjacent frames of the transition frame are determined, and the context information of the action posture corresponding to the adjacent frame is determined; wherein, the context information of the action posture corresponding to the adjacent frame is sampled from the initial state, the target state and the historical information of the action posture of each frame; Based on the context information, determine the change tendency of the transition frame in each dimension and in the latent variable manifold, and calculate the mean and covariance of the context information of the action poses corresponding to the adjacent frames; Based on the mean, covariance, and pose change probability, the adjustment trend of the action pose corresponding to the transition frame is determined, and local feature adjustment is performed on the transition frame according to the adjustment trend to obtain the target frame.

2. The method for generating human motion transitions as described in claim 1, characterized in that, The step of fusing features of the potential state, the initial state, the target state, and the expected state according to the temporal motion prediction model includes: According to the temporal motion prediction model, context information is obtained by sampling from the potential state, the initial state, the target state, or the expected state in a temporal sequence. The expected state is used as the target state when generating each frame of motion pose, and historical information of each frame of motion pose that has been generated is extracted. According to a preset attention mechanism, the context information and the historical information are fused with the action posture corresponding to the expected state as the prediction direction.

3. The method for generating human motion transitions as described in claim 1, characterized in that, The step of mapping the real-time state to a latent variable manifold based on the Gaussian process latent variable model to obtain the latent state corresponding to the probability of change of each action in the real-time state includes: Based on the Gaussian process latent variable model, the pose change features corresponding to each frame of the real-time state are extracted, and the pose change features are stacked into a first data matrix that includes the data dimension of each frame; The real-time states are stacked into a second data matrix with a dimension lower than that of the pose change features; According to the preset action dimensionality reduction mapping function, the first data matrix is ​​embedded into the second data matrix, and the embedded result is mapped to the latent variable manifold to obtain the latent state corresponding to the change probability of each action in the real-time state.

4. The method for generating human motion transitions as described in claim 3, characterized in that, Before the step of embedding the first data matrix into the second data matrix according to the preset action dimensionality reduction mapping function, and mapping the embedded result to the latent variable manifold to obtain the latent state corresponding to the real-time state and the probability of each action change, the following steps are included: Obtain the global dimension control parameters and arbitrary dimension control parameters associated with the potential dimension of the real-time state; Based on the preset control formula, the global dimension control parameters, and the arbitrary dimension control parameters, the preset initial dimensionality reduction mapping function and the preset initial dimensionality increase mapping function are optimized to obtain the preset action dimensionality reduction mapping function and the latent state dimensionality increase mapping function; wherein, the latent state dimensionality increase mapping function is used to transform the latent state corresponding to the latent variable manifold into the actual action of the target human body.

5. The method for generating human motion transitions as described in claim 4, characterized in that, The step of inputting the initial state, the real-time state, the target state, and the expected state into a preset prediction model, and predicting the transition state between the initial state and the target state based on the preset prediction model to obtain the transition frame of the target human body, includes: The initial state, the real-time state, and the target state are input into a preset prediction model, and action features corresponding to the real-time state are extracted based on the preset prediction model; wherein, the action features include at least action speed and action direction; Based on the action speed, the action direction, and the preset prediction model, the transition state between the initial state and the target state is predicted, and the transition frame of the target human body is predicted based on the dimensionality-upgrading mapping function of the transition state and the potential state.

6. The method for generating human motion transitions as described in claim 1, characterized in that, Before the steps of obtaining the initial state, real-time state, target state, and expected state of the target human body, the following are included: The movements of the target human body are captured to obtain a sequence of movement states of the target human body; wherein, each movement in the sequence of movement states is composed of the joint states of the preset root joint and other skeletal joints in the target human body; The sequence of action states is processed into vector parameters to obtain the initial state, real-time state, and target state of the target human body; Each element in the action state sequence is the pose data of the target human body. The pose data includes at least the reference position coordinates of the preset root joint, the velocity parameters of the preset root joint, and the relative position coordinates of other skeletal joints to the preset root joint.

7. A human motion transition generation device, characterized in that, The device includes: a memory, a processor, and a human motion transition generation program stored in the memory and executable on the processor, the human motion transition generation program being configured to implement the steps of the human motion transition generation method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a human motion transition generation program, which, when executed by a processor, implements the steps of the human motion transition generation method as described in any one of claims 1 to 6.

Images