A latent space one-step human action prediction method and device based on a drift model, equipment and medium

Abstract

The application discloses a human body action prediction method and device based on a drift model, and equipment and a medium. The method comprises the following steps: firstly, a variational autoencoder is independently trained to learn a human motion manifold and the parameters thereof are frozen; then, future action latent variables are generated in a latent space by a drift generation model under the condition of historical motion latent variables, and a future human body action sequence is directly obtained after decoding. The application redefines the human body future action prediction as a conditional driving latent space one-step drift generation problem, and the future action latent variables are generated in the latent space by the drift generation model at one time, so that the reasoning speed is improved. Meanwhile, in the training process, the attractive force term and the repulsive force term are calculated according to the weight coefficients determined according to the geometric distance and the semantic similarity, so that the drift direction determined based on the attractive force term and the repulsive force term is jointly modulated by the geometric distance and the action semantic information, and the matching of the prediction result and the real action intention is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for predicting single-step human motion in latent space based on a drift model. Background Technology

[0002] Human motion prediction is an important research direction in the field of artificial intelligence. It aims to predict human posture and movement trajectory in the future based on historically observed human motion sequences. It has wide application value in scenarios such as human-computer interaction, robot collaboration, augmented reality / virtual reality, pedestrian trajectory prediction in autonomous driving, and sports rehabilitation assistance.

[0003] In recent years, significant progress has been made in human motion prediction methods based on deep generative models. One mainstream approach is based on parametric human models or human motion latent spaces, employing generative frameworks such as diffusion models or flow matching models to model future actions. For example, the MLD (Motion Latent-based Diffusion model) method first uses a variational autoencoder to compress human motion sequences into low-dimensional latent variables, and then performs a diffusion process in this latent space to generate motion sequences that meet the conditions, reducing computational redundancy in the original motion space while maintaining high generation quality.

[0004] However, existing methods based on diffusion and flow matching models generally rely on multi-time-step iterative sampling during the inference phase. Each time step requires a forward computation of the neural network, resulting in significant inference latency. In scenarios with extremely high response speed requirements, such as real-time human-computer interaction and real-time obstacle avoidance for robots, this multi-step inference mode is prone to the problem that the actual action has already occurred when the prediction result is output, failing to meet the real-time constraints of human action prediction tasks.

[0005] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0006] The technical problem to be solved by this application is to provide a method, device, equipment and medium for predicting single-step human motion in latent space based on a drift model, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, the first aspect of this application provides a latent space single-step human motion prediction method based on a drift model, wherein the latent space single-step human motion prediction method based on a drift model includes: Construct an initial human motion prediction model, wherein the initial human motion prediction model includes a variational autoencoder and a drift generation model; Historical-future motion sequence pairs are sampled from the training dataset, and the historical-future motion sequence pairs are encoded by a variational autoencoder to obtain historical motion latent variables and future motion latent variables; Using the historical motion latent variables as conditions, the generated motion latent variables and drift field are determined through the drift generation model, and the drift motion latent variables are determined using the drift field and the generated motion latent variables; The initial human motion prediction model is trained based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain a trained human motion prediction model. Predict future human motion sequences using a trained human motion prediction model; The process for determining the drift field is as follows: The weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables are determined based on geometric distance and semantic similarity. Positive samples are future motion latent variables under the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The attractive force term is determined based on the weight coefficients of positive samples, the generated motion latent variables, and all positive samples; the repulsive force term is determined based on the weight coefficients of negative samples, the generated motion latent variables, and all negative samples. The drift field is determined based on the attractive force term and the repulsive force term.

[0008] The aforementioned latent space single-step human motion prediction method based on a drift model, wherein the determination process of the weight coefficients specifically includes: The generated action semantic embedding vector corresponding to the generated motion latent variable and the sample action semantic embedding vector of the target sample are extracted using a pre-trained motion semantic encoder, wherein the target sample is a positive sample or a negative sample. Obtain the geometric distance between the generated motion latent variable and the target sample, and the semantic similarity between the generated action semantic embedding vector and the sample action semantic embedding vector; The geometric distance and the semantic similarity are combined to obtain the weight coefficient.

[0009] The method for predicting single-step human motion in latent space based on a drift model, wherein, after determining the weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables based on geometric distance and semantic similarity, the method further includes: Obtain the maximum semantic similarity between negative samples and all positive samples; The weight coefficients of the negative samples are corrected using a gating weight function based on the maximum semantic similarity.

[0010] The aforementioned latent space single-step human motion prediction method based on a drift model, wherein the step of correcting the weight coefficients of the negative samples based on the maximum semantic similarity using a gating weight function specifically includes: The maximum semantic similarity is compared with the similarity threshold corresponding to the gate weight function; If the maximum semantic similarity is greater than the similarity threshold, then the weight coefficient of the negative sample is reduced. If the maximum semantic similarity is less than or equal to the similarity threshold, the weight coefficient of the negative sample remains unchanged.

[0011] The aforementioned latent space single-step human motion prediction method based on a drift model, wherein training the initial human motion prediction model based on the drift field, the drift motion latent variables, and the future motion latent variables to obtain a trained human motion prediction model specifically includes: A path supervision loss term is constructed based on the drift field and the potential variables of future motion. Construct a drift-supervised loss term based on all generated motion latent variables and all drift motion latent variables in the same training batch; A total loss function is constructed based on the path supervision loss term and the drift supervision loss term, and the initial human motion prediction model is trained using the total loss function to obtain a trained human motion prediction model.

[0012] The latent space single-step human motion prediction method based on a drift model, wherein, before training the initial human motion prediction model based on the drift field, the drift motion latent variables, and the future motion latent variables to obtain a trained human motion prediction model, the method further includes: The variational autoencoder is pre-trained to obtain the model parameters of the trained variational autoencoder. During the training of the initial human motion prediction model based on the generated motion latent variable, the drift motion latent variable, and the future motion latent variable, the model parameters of the trained variational autoencoder are frozen.

[0013] The aforementioned latent space single-step human motion prediction method based on a drift model, wherein the step of predicting future human motion sequences using a trained human motion prediction model specifically includes: The motion encoder in the variational autoencoder is used to map historical human motion sequences into historical motion latent variables. Using the historical motion latent variables as conditions, future motion latent variables are generated step-by-step through a drift generation model; The motion decoder in the variational autoencoder is used to decode the future motion latent variables into a future human motion sequence.

[0014] A second aspect of this application provides a latent space single-step human motion prediction device based on a drift model, wherein the device comprises: A construction module is used to construct an initial human motion prediction model, wherein the initial human motion prediction model includes a variational autoencoder and a drift generation model; The training module is used to sample historical-future motion sequence pairs from the training dataset and encode these pairs using a variational autoencoder to obtain historical motion latent variables and future motion latent variables. Using the historical motion latent variables as conditions, the drift generation model determines generated motion latent variables and a drift field, and the drift field and generated motion latent variables are used to determine drift motion latent variables. Based on the drift field, the drift motion latent variables, and the future motion latent variables, the initial human motion prediction model is trained to obtain a trained human motion prediction model. The drift field is determined based on geometric distance and... The similarity of meaning determines the weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. An attraction term is determined based on the weight coefficients of positive samples, generated motion latent variables, and all positive samples, and a repulsion term is determined based on the weight coefficients of negative samples, generated motion latent variables, and all negative samples. A drift field is determined based on the attraction and repulsion terms. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The inference module is used to predict future human motion sequences using a trained human motion prediction model.

[0015] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the drift model-based latent space single-step human motion prediction method as described above.

[0016] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the latent space single-step human motion prediction method based on the drift model as described above.

[0017] Beneficial effects: 1. This application redefines the prediction of future human actions as a condition-driven latent space single-step drift generation problem. Then, it uses the historical motion latent variables corresponding to the historical human motion sequence as the conditional constraints of the drift generation model. The drift generation model generates the future action latent variables in one step within the latent space. Then, the motion decoder decodes the future action latent variables to obtain the future human action sequence. This allows the generation of future action latent variables to be completed with only a single-step forward computation of the network during inference, which improves the inference speed and significantly reduces the inference latency of human action prediction, thus meeting the real-time requirements of human action prediction tasks.

[0018] 2. In the training process, this application calculates the attraction and repulsion terms based on the weight coefficients determined by geometric distance and semantic similarity. This makes the drift direction determined by the attraction and repulsion terms subject to the joint modulation of geometric distance and action semantic information, thereby improving the matching between the prediction results and the real action intentions. This makes the generated future motion latent variables closer to the real motion distribution and avoids semantic confusion in the generation results of different action categories.

[0019] 3. This application introduces a gating mechanism to correct the repulsion of negative samples based on the maximum semantic similarity, which can make the generated latent variables more consistent with the semantic distribution of the corresponding action category, thereby improving the semantic accuracy and naturalness of the generated human actions.

[0020] 4. This application introduces a path supervision loss term into the total loss term, making full use of the rich single-sample path constraint information in the human motion prediction task, and ensuring the semantic consistency and temporal continuity of each predicted trajectory. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a latent space single-step human motion prediction method based on a drift model provided in an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the framework of the latent space single-step human motion prediction method based on the drift model provided in the embodiments of this application.

[0024] Figure 3 This is a schematic diagram of the semantic enhancement drift field.

[0025] Figure 4This is a schematic diagram of the principle of a drift-based latent space single-step human motion prediction device provided in the embodiments of this application.

[0026] Figure 5 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0027] This application provides a method, apparatus, device, and medium for predicting single-step human motion in latent space based on a drift model. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description, with reference to the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0029] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0030] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0031] Research has shown that human motion prediction is an important research direction in the field of artificial intelligence. It aims to predict human posture and movement trajectory in the future based on historically observed human motion sequences. It has wide application value in scenarios such as human-computer interaction, robot collaboration, augmented reality / virtual reality, pedestrian trajectory prediction in autonomous driving, and sports rehabilitation assistance.

[0032] In recent years, significant progress has been made in human motion prediction methods based on deep generative models. One mainstream approach is based on parametric human models or human motion latent spaces, employing generative frameworks such as diffusion models or flow matching models to model future actions. For example, the MLD (Motion Latent-based Diffusion model) method first uses a variational autoencoder to compress human motion sequences into low-dimensional latent variables, and then performs a diffusion process in this latent space to generate motion sequences that meet the conditions, reducing computational redundancy in the original motion space while maintaining high generation quality.

[0033] However, existing methods based on diffusion and flow matching models generally rely on multi-time-step iterative sampling during the inference phase. Each time step requires a forward computation of the neural network, resulting in significant inference latency. In scenarios with extremely high response speed requirements, such as real-time human-computer interaction and real-time obstacle avoidance for robots, this multi-step inference mode is prone to the problem that the actual action has already occurred when the prediction result is output, failing to meet the real-time constraints of human action prediction tasks.

[0034] Drifting models, as a novel generative modeling paradigm, fundamentally shift the distribution evolution in the generation process from the inference stage to the training stage. During training, a drift field is introduced to drive the distribution of generated samples to gradually approximate the data distribution, allowing generation to be completed with only a single step of network forward computation during inference. However, existing drifting models assign weights to the drift field entirely based on the geometric distance between samples. In human motion prediction scenarios, similar historical postures may lead to completely different future actions (e.g., the same standing posture could result in "walking" or "waving"). This leads to a problem where the predicted results deviate from the actual action intent when applying drifting models to human motion prediction scenarios.

[0035] To address the aforementioned issues, in this embodiment, an initial human motion prediction model is first constructed. Then, historical-future motion sequence pairs are sampled from the training dataset, and these pairs are encoded using a variational autoencoder to obtain historical motion latent variables and future motion latent variables. Using the historical motion latent variables as conditions, a drift generation model is used to determine generated motion latent variables. The weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables are determined based on geometric distance and semantic similarity. Positive samples are future motion latent variables within the same action category in the same training batch, and negative samples are... Generate motion latent variables from the same training batch; determine an attraction term based on the weight coefficients of positive samples, the generated motion latent variables, and all positive samples, and determine a repulsion term based on the weight coefficients of negative samples, the generated motion latent variables, and all negative samples; determine a drift field based on the attraction and repulsion terms, and determine a drift motion latent variable using the drift field and the generated motion latent variables; train the initial human motion prediction model based on the drift field, the drift motion latent variables, and the future motion latent variables to obtain a trained human motion prediction model; finally, use the trained human motion prediction model to predict future human motion sequences. In this embodiment, human future action prediction is redefined as a condition-driven latent space single-step drift generation problem. Historical motion latent variables corresponding to historical human motion sequences are then used as constraints on the drift generation model. The model generates future action latent variables in the latent space in a single pass. A motion decoder then decodes these variables to obtain the future human action sequence. This allows for inference with only a single-step forward computation, improving inference speed and significantly reducing inference latency, thus meeting the real-time requirements of human action prediction tasks. Simultaneously, during training, attractive and repulsive terms are calculated based on weight coefficients determined by geometric distance and semantic similarity. This ensures that the drift direction determined by these terms is jointly modulated by geometric distance and action semantic information, improving the match between the prediction results and the actual action intent. This makes the generated future motion latent variables closer to the actual motion distribution, avoiding semantic confusion between different action categories.

[0036] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0037] This embodiment provides a method for predicting single-step human motion in latent space based on a drift model, such as... Figure 1 and Figure 2 As shown, the latent space single-step human motion prediction method based on the drift model specifically includes: S10. Construct an initial human motion prediction model.

[0038] Specifically, the initial human motion prediction model serves as the initial model for training, and its model parameters are the same as those of the initial model. This initial human motion prediction model includes a variational autoencoder and a drift generation model. The variational autoencoder includes a motion encoder and a motion decoder. The motion encoder is used to encode the human motion sequence into latent variables in the latent space, and the motion decoder is used to decode the latent variables in the latent space to restore the human motion sequence. The drift generation model generates initial future motion latent variables based on historical motion latent variables, and shifts the initial future motion latent variables through a drift field to obtain the future motion latent variables.

[0039] S20. Sample historical-future motion sequence pairs from the training dataset and encode the historical-future motion sequence pairs using a variational autoencoder to obtain historical motion latent variables and future motion latent variables.

[0040] Specifically, the training dataset is a human motion dataset containing multiple human motion sequences of different categories. Each sample in the training dataset is a complete human motion sequence, divided into historical observation segments and future prediction segments according to a time window to obtain historical-future motion sequence pairs. That is, a historical-future motion sequence pair includes a historical human motion sequence and a future human motion sequence, which together constitute a complete human motion sequence. Furthermore, the historical-future motion sequence pair may contain motion category labels, which are semantic annotation information of the motion categories in the historical-future motion sequence pair. Both the historical and future human motion sequences include multiple frames of human posture information. Each frame of human posture includes motion features of a preset number of human joints, which may include joint position, rotation angle, velocity, etc.

[0041] Furthermore, sampling history-future motion sequence pairs from the training dataset involves batch sampling to obtain several history-future motion sequence pairs to form training batches. Then, the historical human motion sequence from each history-future motion sequence pair in the training batch is input into the encoder of the variational autoencoder to obtain the corresponding historical motion latent variable, and the future human motion sequence is input into the encoder of the variational autoencoder to obtain the corresponding future motion latent variable. All historical and future motion latent variables are distributed in the latent space learned by the variational autoencoder. The encoding process of the encoder in the variational autoencoder can be described as mapping the human motion sequence to Gaussian distribution parameters (i.e., mean and variance) in the latent space, and then sampling the motion latent variables from these Gaussian distribution parameters through reparameterization.

[0042] S30. Using the historical motion latent variables as conditions, determine the generated motion latent variables and the drift field through the drift generation model, and use the drift field and the generated motion latent variables to determine the drift motion latent variables.

[0043] Specifically, the drift generation model uses historical motion latent variables as input conditions. It performs a one-step forward computation on initial noise latent variables in the latent space (such as pure noise states sampled from a standard Gaussian distribution), directly outputting the generated motion latent variables and drift field under these conditions. Then, the drift field is used to offset the generated motion latent variables to determine the drift motion latent variables, which serve as the future motion latent variables predicted by the drift generation model. Compared to the multi-step iterative sampling of traditional diffusion models, this process requires only one neural network inference to obtain the future motion latent variables, eliminating the need for multiple rounds of repeated computation and reducing inference time in the generation process.

[0044] In one embodiment, the drift generation model can employ a Transformer-based architecture. Initial noise latent variables are linearly projected and used as the sequence input to the Transformer encoder. Historical motion latent variables are injected into each Transformer block of the encoder through an adaptive layer normalization mechanism. Within each Transformer block, the historical motion latent variables are linearly transformed to generate scaling and offset parameters. These scaling and offset parameters are then used to modulate the intermediate features of each layer in that Transformer block, ensuring that each layer is guided by the historical human motion sequence. Of course, in practical applications, the drift generation model can employ other network architectures, such as one-dimensional convolutional networks or MLP-Mixer networks. Historical motion latent variables can also be injected into the network layers of the drift generation model in other ways, such as using cross-attention mechanisms or feature concatenation followed by projection.

[0045] In one embodiment, the process of determining the drift field is as follows: H10. Determine the weight coefficients of the positive samples corresponding to the generated motion latent variables and the weight coefficients of the negative samples corresponding to the generated motion latent variables based on geometric distance and semantic similarity. H20, based on the weight coefficients of positive samples, generate motion latent variables and determine the attraction term based on all positive samples, and based on the weight coefficients of negative samples, generate motion latent variables and determine the repulsion term based on all negative samples; H30. Determine the drift field based on the attractive force term and the repulsive force term.

[0046] In step H10, positive samples are future motion latent variables under the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The construction process for positive and negative samples can be as follows: Cluster the historical-future motion sequence pairs in the training batch according to action category labels to obtain several sample groups. Then, use the future motion latent variables of other historical-future motion sequence pairs in the sample group to which the currently input historical-future motion sequence pair belongs as positive samples, thus obtaining all positive samples for the currently input historical-future motion sequence pair. Then, use the generated motion latent variables formed by all historical-future motion sequence pairs in the training batch as negative samples, thus obtaining all negative samples for the currently input historical-future motion sequence pair. When the historical-future motion sequence pair does not contain action category labels, the historical motion latent variables can be clustered to form several sample groups.

[0047] Furthermore, in practical applications, when the number of positive samples in the training batch is insufficient (e.g., less than the preset number), data augmentation techniques such as time shifting, random pruning, and joint noise perturbation can be applied to the same historical-future motion sequence pair to generate additional positive samples.

[0048] Furthermore, after obtaining positive and negative samples, the weight coefficients of positive samples can be determined by calculating the geometric distance and semantic similarity between the generated latent motion variables and the positive samples, and the weight coefficients of negative samples can be determined by calculating the geometric distance and semantic similarity between the generated latent motion variables and the negative samples. Specifically, the process of determining the weight coefficients includes: The generated action semantic embedding vectors corresponding to the generated motion latent variables and the sample action semantic embedding vectors of the target samples are extracted using a pre-trained motion semantic encoder. Obtain the geometric distance between the generated motion latent variable and the target sample, and the semantic similarity between the generated action semantic embedding vector and the sample action semantic embedding vector; The geometric distance and the semantic similarity are combined to obtain the weight coefficient.

[0049] Specifically, since the latent space of a variational autoencoder (VAE) is optimized for reconstruction accuracy, its geometry reflects the similarity of motion details (such as joint angles and posture shapes) rather than semantic consistency at the action level. For example, the first few frames of "walking slowly" and "starting quickly" may be geometrically close in the VAE (e.g., similar initial postures), but semantically completely different; conversely, "waving" actions at different speeds and amplitudes may be far apart in the VAE's latent space, but semantically highly consistent. Therefore, this application introduces a pre-trained motion semantic encoder, which extracts motion semantic embedding vectors that express the action semantic information from the input motion latent variables. This provides an action-level similarity metric orthogonal to the VAE reconstruction space, enabling the drift field to distinguish between semantic and geometric ambiguities. It can be understood that the motion latent variables are generated by the motion semantic encoder to extract the motion semantic embedding vectors, and the target samples are extracted by the motion semantic encoder to extract the sample motion semantic embedding vectors. These target samples can be positive or negative.

[0050] In the pre-training process of the motion semantic encoder, the encoder uses action category labels provided by the human motion dataset with labeled action categories as supervision signals. Through contrastive learning, motion latent variables belonging to the same action category (such as "walking," "sitting," "waving," etc.) are brought closer together in the semantic space, while motion latent variables belonging to different action categories are moved further apart. Then, during the training and inference processes of the human motion prediction model, the model parameters of the motion semantic encoder are frozen. Only the motion latent variables are needed as input to extract the action semantic embedding vectors, without relying on any text information. Of course, in practical applications, the motion semantic encoder can also be trained in other ways. For example, it can be pre-trained under unlabeled conditions using self-supervised learning methods (such as contrastive prediction encoding on motion data) to automatically discover the clustering structure of motion patterns; or, when the dataset includes text descriptions, the text encoder and motion encoder can be jointly trained to share an embedding space.

[0051] Furthermore, after obtaining the generated action semantic embedding vector corresponding to the generated motion latent variable and the sample action semantic embedding vector of the target sample, the geometric distance between the generated motion latent variable and the target sample, as well as the semantic similarity between the generated action semantic embedding vector and the sample action semantic embedding vector, can be obtained. The geometric distance can be determined using a geometric distance kernel, which measures the distance between two motion latent variables in the latent space; the semantic similarity can be determined using a semantic similarity kernel, which measures the cosine similarity of the action semantic embedding vectors in the action semantic space.

[0052] like Figure 3As shown, after obtaining geometric distance and semantic similarity, the weight coefficients can be determined by combining them. Specifically, this can be achieved by multiplying the geometric distance and semantic similarity scores; that is, by multiplying the geometric distance kernel and the semantic similarity kernel to construct a semantic enhancement kernel function, which can then be used to determine the weight coefficients for positive and negative samples. Furthermore, in practical applications, document parameters can be configured for geometric distance and semantic similarity to control their sensitivity. Of course, in practical applications, geometric distance and semantic similarity can be combined in other ways, such as weighted summation or attention-based fusion methods.

[0053] This application's embodiments introduce semantic similarity when determining weighting coefficients, so that the weighting coefficients of attractive and repulsive forces in the drift field simultaneously reflect geometric proximity and semantic consistency of actions, resulting in similar motion shapes and actions. Figure 1 Positive samples with similar intentions generate stronger attraction, while samples with different action intentions, even if they are geometrically close, will not generate misleading strong attraction. This makes the drift direction determined based on attraction terms and attraction terms jointly modulated by geometric distance and action semantic information, improving the matching of prediction results with real action intentions. This makes the generated future motion latent variables closer to the real motion distribution and avoids semantic confusion in the generated results of different action categories.

[0054] In one embodiment, in the application of drift generation models to human motion prediction, it was found that the repulsive force term tends to dominate the optimization direction in the early stages of training. Since the future movements of the human body inherently involve one-to-many uncertainty, excessively strong repulsive forces can prematurely push potentially reasonable generation results away from the correct region, causing training instability. Therefore, this embodiment introduces a gating mechanism based on the semantic similarity of positive and negative samples, through which the weight coefficients of negative samples are adaptively adjusted. Correspondingly, after determining the weight coefficients of the positive and negative samples corresponding to the generated motion latent variables based on geometric distance and semantic similarity, the method further includes: Obtain the maximum semantic similarity between negative samples and all positive samples; The weight coefficients of the negative samples are corrected using a gating weight function based on the maximum semantic similarity.

[0055] Specifically, for each negative sample, the semantic similarity between the negative sample and all positive samples of the currently generated motion latent variable is calculated, and the maximum semantic similarity is taken. This maximum semantic similarity reflects the semantic closeness of the negative sample to the action category corresponding to the currently generated motion latent variable: if the negative sample belongs to the action category corresponding to the currently generated motion latent variable, the maximum semantic similarity will be higher; if it belongs to a different action category, the maximum semantic similarity will be lower. Then, the maximum semantic similarity is input into a preset gating weight function, and the weight coefficients of the negative sample are corrected through the gating weight function to obtain the corrected negative sample weight coefficients.

[0056] The gating weight function is used to adjust the weight coefficients of negative samples, ensuring that negative samples with high semantic similarity correspond to smaller adjusted weight coefficients. This applies a smaller repulsive force to negative samples that are semantically close to the action category corresponding to the currently generated motion latent variable, preventing reasonable future motion latent variables from being pushed away. Conversely, it applies a full repulsive force to negative samples that are semantically significantly different from the action category corresponding to the currently generated motion latent variable, ensuring semantic discrimination ability and effectively improving the stability of the training process. This prevents the repulsive force term from excessively dominating the optimization in the early stages of training, which could lead to model convergence failure. Furthermore, as training progresses and the generated distribution gradually approaches the true distribution, the regulatory effect of the gating mechanism naturally weakens, without affecting the final distribution matching accuracy.

[0057] Based on this, in one embodiment, the step of correcting the weight coefficients of the negative samples according to the maximum semantic similarity through a gating weight function specifically includes: The maximum semantic similarity is compared with the similarity threshold corresponding to the gate weight function; If the maximum semantic similarity is greater than the similarity threshold, then the weight coefficient of the negative sample is reduced. If the maximum semantic similarity is less than or equal to the similarity threshold, the weight coefficient of the negative sample remains unchanged.

[0058] Specifically, the similarity threshold can be preset according to the actual application scenario, for example, it can be set to values ​​such as 0.5 or 0.8. The reduction of the weight coefficient of the negative sample can be achieved by directly adjusting the weight coefficient of the negative sample to a preset weight coefficient (such as a value close to 0), or by determining the adjustment coefficient according to the preset correspondence between the maximum semantic similarity and the adjustment coefficient, and then using this adjustment coefficient to reduce the weight coefficient of the negative sample. The correspondence between the maximum semantic similarity and the adjustment coefficient is an increasing relationship, so that the greater the maximum semantic similarity of the negative sample, the smaller the weight coefficient of the negative sample.

[0059] Of course, in practical applications, the gating weight function can also take other forms, such as hard threshold truncation, Gaussian gating, and retaining only the top K negative samples with the lowest interference.

[0060] In step H20, after obtaining the weight coefficients of positive and negative samples, the attractive force term can be determined by taking a weighted average of the offset vectors of the positive samples and the generated latent motion variables based on the weight coefficients of the positive samples. That is, the attractive force term is the weighted average offset vector between the generated latent motion variables and all positive samples. Similarly, the repulsive force term can be determined by taking a weighted average of the offset vectors of the negative samples and the generated latent motion variables based on the weight coefficients of the negative samples. That is, the repulsive force term is the weighted average offset vector between the generated latent motion variables and all negative samples. The calculation method of the repulsive force term is symmetrical to that of the attractive force term, ensuring mutual repulsion among the generated samples and preventing distribution collapse.

[0061] In step H30, the drift field is the attractive force term minus the repulsive force term. When the generated distribution perfectly matches the data distribution, the attractive and repulsive forces cancel each other out, and the drift field returns to zero to reach an equilibrium state. Furthermore, after obtaining the drift motion latent variable, this drift motion latent variable is used as the frozen target motion latent variable, and then the generated motion latent variable is brought closer to this target motion latent variable using a loss function.

[0062] S40. The initial human motion prediction model is trained based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain a trained human motion prediction model.

[0063] Specifically, the initial human motion prediction model is trained to approximate the overall data distribution of the generated distribution. In the human motion prediction task, in addition to distribution matching supervision, there are also abundant single-sample-level path constraints. Therefore, in this embodiment, when training the initial human motion prediction model, a total loss function is constructed based on the drift field, the drift motion latent variable, and the future motion latent variable. The total loss function is used for single-sample-level path constraints and distribution matching supervision; that is, the total loss function includes a path supervision loss term and a drift supervision loss term. The path supervision loss term is used for single-sample-level path constraints, and the drift supervision loss term is used for distribution matching supervision.

[0064] In one embodiment, training the initial human motion prediction model based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain a trained human motion prediction model specifically includes: A path supervision loss term is constructed based on the drift field and the potential variables of future motion. Construct a drift-supervised loss term based on all generated motion latent variables and all drift motion latent variables in the same training batch; A total loss function is constructed based on the path supervision loss term and the drift supervision loss term, and the initial human motion prediction model is trained using the total loss function to obtain a trained human motion prediction model.

[0065] Specifically, the path supervision loss term measures the distance between generated motion latent variables and their corresponding future motion latent variables, emphasizing the accuracy of the mapping from specific input noise to the target latent variable, ensuring that each predicted path points to a semantically correct future action target. However, when constructing the path supervision loss term for each generated motion latent variable, it is not required that all noise inputs converge to the same future motion latent variable under the same historical conditions. Therefore, in each training iteration, a historical-future motion sequence pair corresponding to the generated motion latent variable in the same training batch is bound, so that the future motion latent variable used to determine the path supervision loss term is the future motion latent variable determined by a historical-future motion sequence pair in the same training batch. Furthermore, the generated motion latent variables of the same historical-future motion sequence pair in different training batches are associated with different generated motion latent variables, ensuring that the drift generation model receives diverse directional signals. The path supervision loss term can be constructed based on the drift field and the true offset field between the future motion latent variable and its corresponding initial noise variable. For example, the true offset field between the future motion latent variable and the initial noise variable can be calculated first, and then the loss between the drift field and the true offset field can be calculated using methods such as mean squared error loss to obtain the path supervision loss term.

[0066] The drift supervision loss term represents the drift loss between the generated latent motion variables and all drifting latent motion variables. This term drives the overall generation distribution to approximate the conditionally true distribution. The drift supervision loss term preserves the diversity and distributional rationality of future motions. The drift supervision loss term can employ methods such as mean squared error loss.

[0067] Furthermore, after obtaining the path supervision loss term and the drift supervision loss term, they are fused into a total loss function using weight coefficients. In actual training, a fixed weight (e.g., both have equal weight) or a dynamic weight adjustment strategy can be adopted (path supervision is dominant in the early stages of training, and the weight of distributed supervision is gradually increased as training progresses). In this embodiment, the weight coefficients of the path supervision loss term are adjusted at different training stages using a dynamic weight adjustment strategy, so that the path supervision loss term mainly plays a coarse-grained directional guidance role in the early stages of training, while distributed supervision takes over in the later stages of training to ensure multi-peak coverage capability.

[0068] The embodiments of this application adopt a hybrid framework of path supervision loss and drift supervision loss. The path supervision loss ensures that the model learns "where to predict" (semantic direction correctness), while the drift supervision loss ensures that the model learns "what kind of reasonable future distribution the overall prediction result should fall on" (distribution rationality). This allows the drift generation model to take into account generation quality, semantic alignment and distribution diversity while achieving efficient single-step reasoning.

[0069] Furthermore, in one embodiment, this application employs a two-stage training process. The first training stage involves pre-training the variational autoencoder (VAE), and the second training stage involves freezing the VAE model parameters and training the drift generation model. Based on this, before sampling historical-future motion sequence pairs from the training dataset and encoding these pairs using the VAE to obtain historical and future motion latent variables, the method further includes pre-training the VAE to obtain the trained VAE model parameters. Then, during the training of the initial human motion prediction model based on the generated motion latent variables, the drift motion latent variables, and the future motion latent variables, the trained VAE model parameters are kept frozen.

[0070] S80. Predict future human motion sequences using a trained human motion prediction model.

[0071] Specifically, after obtaining a trained human motion prediction model, the model is used to predict future human motion sequences. That is, after obtaining historical human motion sequences, these sequences are input into the trained human motion prediction model, which then outputs future human motion sequences. The specific process of predicting future human motion sequences using the trained human motion prediction model can be as follows: The motion encoder in a variational autoencoder maps the historical human motion sequences to historical motion latent variables; using these historical motion latent variables as conditions, a drift generation model generates future motion latent variables step-by-step; and the motion decoder in the variational autoencoder decodes these future motion latent variables into future human motion sequences.

[0072] In this embodiment, the human motion prediction model trained during inference only requires one forward computation of the generator network, eliminating the need for multi-step iterative denoising or sampling, resulting in extremely low inference latency. Compared to existing prediction methods with dozens of diffusion inference steps (such as the MLD method), this embodiment improves inference efficiency by approximately 7 times, meeting the stringent latency requirements of real-time human-computer interaction and robot collaboration scenarios.

[0073] Based on the aforementioned method for predicting single-step human motion in latent space using a drift model, this embodiment provides a method for predicting single-step human motion in latent space using a drift model, such as... Figure 4 As shown, the device includes: The construction module 100 is used to construct an initial human motion prediction model, wherein the initial human motion prediction model includes a variational autoencoder and a drift generation model. Training module 200 is used to sample historical-future motion sequence pairs from the training dataset and encode these pairs using a variational autoencoder to obtain historical motion latent variables and future motion latent variables. Using the historical motion latent variables as conditions, a drift generation model is used to determine generated motion latent variables and a drift field, and the drift field and generated motion latent variables are used to determine drift motion latent variables. Based on the drift field, the drift motion latent variables, and the future motion latent variables, the initial human motion prediction model is trained to obtain a trained human motion prediction model. The drift field is determined based on geometric distance and... Semantic similarity determines the weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. An attraction term is determined based on the weight coefficients of positive samples, generated motion latent variables, and all positive samples, and a repulsion term is determined based on the weight coefficients of negative samples, generated motion latent variables, and all negative samples. A drift field is determined based on the attraction and repulsion terms. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The inference module 300 is used to predict future human motion sequences using a trained human motion prediction model.

[0074] Based on the above-described method for predicting single-step human motion in latent space using a drift model, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the method for predicting single-step human motion in latent space using a drift model as described in the above embodiment.

[0075] Based on the aforementioned method for predicting single-step human motion in latent space using a drift model, this application also provides a terminal device, such as... Figure 5As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0076] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0077] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0078] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0079] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A latent space single-step human motion prediction method based on a drift model, characterized in that, The latent space single-step human motion prediction method based on the drift model includes: Construct an initial human motion prediction model, wherein the initial human motion prediction model includes a variational autoencoder and a drift generation model; Historical-future motion sequence pairs are sampled from the training dataset, and the historical-future motion sequence pairs are encoded by a variational autoencoder to obtain historical motion latent variables and future motion latent variables; Using the historical motion latent variables as conditions, the generated motion latent variables and drift field are determined through the drift generation model, and the drift motion latent variables are determined using the drift field and the generated motion latent variables; The initial human motion prediction model is trained based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain a trained human motion prediction model. Predict future human motion sequences using a trained human motion prediction model; The process for determining the drift field is as follows: The weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables are determined based on geometric distance and semantic similarity. Positive samples are future motion latent variables under the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The attractive force term is determined based on the weight coefficients of positive samples, the generated motion latent variables, and all positive samples; the repulsive force term is determined based on the weight coefficients of negative samples, the generated motion latent variables, and all negative samples. The drift field is determined based on the attractive force term and the repulsive force term.

2. The latent space single-step human motion prediction method based on a drift model according to claim 1, characterized in that, The process of determining the weighting coefficients specifically includes: The generated action semantic embedding vector corresponding to the generated motion latent variable and the sample action semantic embedding vector of the target sample are extracted using a pre-trained motion semantic encoder, wherein the target sample is a positive sample or a negative sample. Obtain the geometric distance between the generated motion latent variable and the target sample, and the semantic similarity between the generated action semantic embedding vector and the sample action semantic embedding vector; The geometric distance and the semantic similarity are combined to obtain the weight coefficient.

3. The latent space single-step human motion prediction method based on a drift model according to claim 1 or 2, characterized in that, After determining the weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables based on geometric distance and semantic similarity, the method further includes: Obtain the maximum semantic similarity between negative samples and all positive samples; The weight coefficients of the negative samples are corrected using a gating weight function based on the maximum semantic similarity.

4. The latent space single-step human motion prediction method based on a drift model according to claim 3, characterized in that, The step of correcting the weight coefficients of the negative samples based on the maximum semantic similarity using a gated weight function specifically includes: The maximum semantic similarity is compared with the similarity threshold corresponding to the gate weight function; If the maximum semantic similarity is greater than the similarity threshold, then the weight coefficient of the negative sample is reduced. If the maximum semantic similarity is less than or equal to the similarity threshold, the weight coefficient of the negative sample remains unchanged.

5. The latent space single-step human motion prediction method based on a drift model according to claim 1, characterized in that, The step of training the initial human motion prediction model based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain the trained human motion prediction model specifically includes: A path supervision loss term is constructed based on the drift field and the potential variables of future motion. Construct a drift-supervised loss term based on all generated motion latent variables and all drift motion latent variables in the same training batch; A total loss function is constructed based on the path supervision loss term and the drift supervision loss term, and the initial human motion prediction model is trained using the total loss function to obtain a trained human motion prediction model.

6. The latent space single-step human motion prediction method based on a drift model according to claim 1, characterized in that, Before training the initial human motion prediction model based on the drift field, the drift motion latent variable, and the future motion latent variable to obtain the trained human motion prediction model, the method further includes: The variational autoencoder is pre-trained to obtain the model parameters of the trained variational autoencoder. During the training of the initial human motion prediction model based on the generated motion latent variable, the drift motion latent variable, and the future motion latent variable, the model parameters of the trained variational autoencoder are frozen.

7. The latent space single-step human motion prediction method based on a drift model according to claim 1, characterized in that, The method of predicting future human motion sequences using a trained human motion prediction model specifically includes: The motion encoder in the variational autoencoder is used to map historical human motion sequences into historical motion latent variables. Using the historical motion latent variables as conditions, future motion latent variables are generated step-by-step through a drift generation model; The motion decoder in the variational autoencoder is used to decode the future motion latent variables into a future human motion sequence.

8. A latent space single-step human motion prediction device based on a drift model, characterized in that, The device includes: A construction module is used to construct an initial human motion prediction model, wherein the initial human motion prediction model includes a variational autoencoder and a drift generation model; The training module is used to sample historical-future motion sequence pairs from the training dataset and encode these pairs using a variational autoencoder to obtain historical motion latent variables and future motion latent variables. Using the historical motion latent variables as conditions, the drift generation model determines generated motion latent variables and a drift field, and the drift field and generated motion latent variables are used to determine drift motion latent variables. Based on the drift field, the drift motion latent variables, and the future motion latent variables, the initial human motion prediction model is trained to obtain a trained human motion prediction model. The drift field is determined based on geometric distance and... The similarity of meaning determines the weight coefficients of positive samples and negative samples corresponding to the generated motion latent variables. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. An attraction term is determined based on the weight coefficients of positive samples, generated motion latent variables, and all positive samples, and a repulsion term is determined based on the weight coefficients of negative samples, generated motion latent variables, and all negative samples. A drift field is determined based on the attraction and repulsion terms. Positive samples are future motion latent variables of the same action category in the same training batch, and negative samples are generated motion latent variables in the same training batch. The inference module is used to predict future human motion sequences using a trained human motion prediction model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the drift model-based latent space single-step human motion prediction method as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the latent space single-step human motion prediction method based on the drift model as described in any one of claims 1-7.

Images