Multi-condition human body action generation method and system based on scene and text

Abstract

The invention provides a scene and text-based multi-condition human body action generation method and system, and the method comprises the steps: firstly obtaining a ternary data set, carrying out the preprocessing of the ternary data set, and carrying out the feature extraction of scene data and text data, and obtaining scene features and text features; performing noise addition processing on the action data in the preprocessed ternary data set, and inputting the action data into a de-noising device in combination with scene features and text features to obtain de-noised action data; jointly training a scene encoder and a de-noising device by using the action data and the de-noised action data; and finally, the user generates new action data by using the trained scene encoder and the de-noising device. The bounding box is used for representing the scene, and scene information can be quickly and efficiently captured; meanwhile, by fusing the scene data and the text description, the generated action not only meets the semantic requirement of the text description, but also can reasonably interact with a specific scene, and the stability and accuracy of the generated action are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer graphics, three-dimensional digital human and motion generation, and more specifically, to a method and system for generating multi-conditional human motion based on scenes and texts. Background Art

[0002] Human motion generation models are widely used in fields such as virtual reality, animation, game development, and human-computer interaction. Such models can automatically generate high-quality motions that meet specific needs based on multiple input conditions such as voice, text, and scenes, significantly improving production efficiency, enhancing user experience, and promoting research progress in technologies such as motion recognition and posture estimation. In particular, in the field of automated motion generation, how to accurately combine semantic information with environmental constraints has always been a research focus.

[0003] Existing generative models usually require input of action data, text descriptions, and scene information. Action data is mostly represented by a sequence of human key points or morphological parameters (such as the SMPL / SMPL-X model); text input is usually a sequence of words or text that describes the type, details, and context of the action; and scene data is mostly in the form of point cloud data or bounding boxes, providing physical and spatial constraints. One of the key challenges of existing technologies is how to process and fuse these heterogeneous data to generate natural actions that conform to semantics and scenes.

[0004] Most current research relies on large-scale action-text-scene datasets, the size and quality of which directly affect the generalization ability of the generative model. Generalization ability refers to the ability of the model to generate reasonable actions when faced with text and scenes that are not seen or rare in the dataset. Due to the limitations of the dataset, many generative models find it difficult to effectively handle complex and dynamically changing scene information, especially in the actual application of action generation, where the diversity and complexity of scene conditions are often not fully considered.

[0005] In addition, although some methods try to enhance the robustness of generative models by introducing techniques such as self-supervised learning and generative adversarial networks, these methods usually face difficulties in fusing scene and text information due to the high correlation between scenes and actions. Therefore, how to accurately integrate multimodal information and improve the quality and adaptability of action generation remains an important research direction in this field.

[0006] At present, a patent application with publication number "CN117953113A" discloses a method for generating human motion that can interact with three-dimensional scene targets and users. This technology extracts and encodes features of scenes, semantics and human motions by inputting scenes, text descriptions and training action sequences, and fuses these features into an iteratively optimized trajectory decoder; using a dual Transformer structure, this method iteratively learns path planning and text-controlled human motion sequences in scenes, generates 3D motion sequences corresponding to user needs and scenes, and realizes controllable text scene motion generation for human-computer scene interaction; the disadvantage of this technology is that the fusion process of scene and text features may not be sufficient to capture complex scene constraints and semantic relationships, which may result in the generated motion lacking physical rationality and semantic accuracy, and the generalization ability for new scenes and diversified descriptions may be limited.

[0007] The patent application with publication number "CN118644895A" also discloses a text-conditional human motion generation method based on a discrete diffusion model; the technology includes four steps: first, obtaining a human 3D key point data set; then, using the data set to pre-train the motion quantization variational autoencoder to obtain motion latent variables; then, using the human 3D key point data set and the motion latent variables to train the lightweight discrete diffusion model; finally, inputting the given text condition into the text encoder to generate a text feature vector, and decoding the 3D key point motion sequence corresponding to the text condition through the trained diffusion model and variational autoencoder; the disadvantage of this technology is that although it can generate high-precision actions that conform to the text description, it cannot consider the scene as a condition, and only relies on text features to generate actions. It lacks understanding of the 3D scene environment, so when generating actions that interact with complex scenes, it lacks physical rationality and environmental adaptability.

[0008] In addition, the public paper "Scaling up dynamic human-scene interaction modeling, CVPR2024" proposed a human motion generation method based on an autoregressive action diffusion model; this technology first gradually generates a motion sequence through a conditional diffusion model based on the 3D scene and target position; then, the overall goal is subdivided into smaller sub-goals to ensure precise control of the implementation of the action in each action segment; at the same time, frame-by-frame action labels are introduced to allow the action to evolve naturally in different segments, enhancing the coherence and realism of the sequence; finally, by optimizing the interaction with dynamic objects, the generated human action sequence is ensured to match the dynamics of the environment; the disadvantage of this technology is that text cannot be used for interaction, which limits the flexibility of the user. Secondly, the subdivision of sub-goals and frame-by-frame action labels makes the operation more complicated and increases the learning cost of the user; in addition, although real-time generation is supported, the computational requirements may cause the generation speed to slow down when dealing with complex scenes. Summary of the invention

[0009] In order to overcome the defects of the above-mentioned prior art in lacking physical rationality and environmental adaptability when generating actions to interact with complex scenes, the present invention provides a multi-conditional human action generation method and system based on scenes and texts. By using bounding boxes to represent scenes, scene information can be captured quickly and efficiently. At the same time, by fusing scene data and text descriptions, the present invention generates actions that not only meet the semantic requirements of text descriptions, but can also reasonably interact with specific scenes, effectively improving the accuracy of generated actions.

[0010] In order to solve the above technical problems, the technical solution of the present invention is as follows: A method for generating multi-conditional human motion based on scenes and texts comprises the following steps: S1: obtaining a ternary data set including initial action data, scene data and text data, performing bounding box extraction on the scene data, and performing dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; S2: Construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features and text features ; S3: Build denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; S4: Using the pre-processed action data in the triple data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; S5: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

[0011] Preferably, in step S1, the ternary data set includes a plurality of ternary data samples, each of which includes an initial action data, a scene data and a text data; The initial action data is specifically a frame sequence of key points of the human body; the scene data is specifically a point cloud of each object in the scene; and the text data is specifically a text description of the action data and the scene data.

[0012] Preferably, in step S1, initial motion data including several motions is obtained, and the data dimension of a single motion is (F, 24, 3), wherein F is the number of motion frames, each frame contains a static posture of the human body, and the static posture is represented by the three-dimensional coordinates of 24 human key points, and 3 indicates that the position of each human key point is represented by three-dimensional space coordinates; the data dimensions of all motions are uniformly filled to a fixed number of frames using a zero-filling method. , and generate a mask at the same time, complete the dimension filling of the initial action data, and obtain the action data ; For the scene data in the ternary data set, the bounding boxes of each object in the scene are extracted; each of the bounding boxes represents an object, and the parameters of the i-th bounding box include shape information and category information ,in, Represents the center coordinates of the i-th bounding box; Indicates the size of the i-th bounding box; Represents the rotation angle of the i-th bounding box around the vertical axis; The scene data after bounding box extraction contains the bounding boxes of N objects. The zero-filling method is used to uniformly fill the scene data after bounding box extraction to a fixed number. , complete the dimension filling of the scene data after the bounding box is extracted.

[0013] Preferably, in step S2, the constructed scene encoder is based on a Transformer encoder structure, and the initial parameters are random parameters; For the bounding box of the i-th object , use two parallel multi-layer perceptrons to process the shape information of the i-th bounding box respectively and category information , get shape features and category features; concatenate the shape features and category features to get the i-th bounding box The eigenvector of ; For multiple bounding box features in scene data , using the Transformer encoder to perform scene-level global feature aggregation; the feature vector of each bounding box is aggregated through the self-attention mechanism in the Transformer encoder to capture the relationship between objects and global context information, and finally obtain the scene feature containing global information ; The text data is encoded using the text encoder in the pre-trained CLIP model to obtain the text features .

[0014] Preferably, step S3 comprises the following steps: Randomly sample time steps from a uniform distribution on the interval [0,T] , and according to the time step Action data Perform noise processing to generate noisy motion data ,in, Represents action data The i-th frame in Represents the action data after adding noise The i-th frame in; The formula for the noise addition process is as follows:

[0015] in, is standard Gaussian noise; is a predefined time step Decreasing noise weight function; The scene features and text features The dimensions are aligned through a linear layer respectively, and the two features after dimension alignment are added and fused to obtain the joint feature. , expressed as:

[0016] For the noise-added motion data Each frame in , using a multi-layer perceptron to process each frame As an action code , a total of action code units, represented by ; The time step is calculated using the following formula: Perform cosine position encoding:

[0017]

[0018] in, and Represents the time step Encoded values ​​on even and odd dimensions, is the dimension index, is the preset embedding dimension; The cosine position code is combined with the joint feature Add together to get the conditional code ; The denoiser is based on the Transformer encoder structure, and the initial parameters are random parameters; As the input of the denoiser, the output of the denoiser is the denoised codeword sequence , the action codewords in the denoised codeword sequence Through a linear layer, we get the denoised action data .

[0019] Preferably, in step S4, the preset loss function is specifically a mean square error loss function, and the calculation formula is:

[0020] in, is the mean square error loss function value; is the total number of triple data samples in the triple data set; is the i-th action data ; is the denoised action data of the i-th item .

[0021] Preferably, step S5 comprises the following steps: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features ; The initial noise generated by random sampling , and new scene features and new text features input the trained denoiser together; The trained denoiser obtains the action data after the first round of iteration after the first round of denoising. ; Then for the time step Perform the noise addition operation to obtain the noise in the second round of iteration ; Then, the trained denoiser is used to perform a second round of denoising to obtain the action data after the second round of iteration ; Repeat the above process and use the trained denoiser to After a round of iteration, we finally get Action data after round iteration , and as the new action data , expressed as: .

[0022] The present invention also provides a scene and text-based multi-conditional human motion generation system, which applies the scene and text-based multi-conditional human motion generation method, including: A data preprocessing unit is used to obtain a ternary data set including initial action data, scene data and text data, perform bounding box extraction on the scene data, and perform dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; Feature extraction unit: used to construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features. and text features ; Prediction result generation unit: used to build a denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; Joint training unit: used to utilize the action data in the preprocessed ternary data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; Action generation unit: used to obtain new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

[0023] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method are implemented.

[0024] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the above method are executed.

[0025] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention provides a method and system for generating multi-conditional human motion based on scenes and texts, and the main problem to be solved is: how to combine scene data with text descriptions and generate motions using a denoising diffusion model to ensure that the generated motions meet the text semantic requirements and are closely integrated with scene information; the steps of the method are: firstly, a ternary data set is acquired and preprocessed, then feature extraction is performed on the scene data and text data to obtain scene features and text features; then, noise is added to the motion data in the preprocessed ternary data set, and the scene features and text features are combined and input into a denoiser to obtain denoised motion data; the motion data and the denoised motion data are used to jointly train a scene encoder and a denoiser; finally, a user generates new motion data using the trained scene encoder and denoiser; The present invention has the following advantages: 1) By using bounding boxes to represent scenes, the present invention can quickly and efficiently capture scene information, which is particularly suitable for real-time interaction requirements of dynamic scenes such as games. As a concise scene representation method, bounding boxes have lower computing overhead and higher processing efficiency compared to complex 3D models or point cloud data. 2) The present invention integrates scene and text conditions. The generated actions not only meet the semantic requirements of text description, but also can reasonably interact with specific scenes. By extracting scene features and text features through scene encoder and text encoder respectively, the model can accurately understand environmental constraints and action details, thereby generating action sequences with more physical rationality and scene adaptability. 3) The present invention adopts a denoising diffusion model for progressive denoising, ensuring the stability and accuracy of action generation; compared with traditional methods, the present invention can effectively combine scene and text information, avoids the limitation of ignoring environmental constraints, and improves the versatility of the model, which is particularly suitable for real-time interaction in the fields of virtual reality, game development, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an overall flow chart of a multi-conditional human motion generation method based on scenes and texts provided in Example 1.

[0027] Figure 2 This is a specific implementation flow chart of a multi-conditional human motion generation method based on scenes and texts provided in Example 2.

[0028] Figure 3 This is a schematic diagram of the denoising process provided in Example 2.

[0029] Figure 4 This is a structural diagram of a scene and text-based multi-conditional human motion generation system provided in Example 3. DETAILED DESCRIPTION

[0030] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention; In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0031] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0032] Example 1 like Figure 1 As shown, this embodiment provides a multi-conditional human action generation method based on scenes and texts, comprising the following steps: S1: obtaining a ternary data set including initial action data, scene data and text data, performing bounding box extraction on the scene data, and performing dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; S2: Construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features and text features ; S3: Build denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; S4: Using the pre-processed action data in the triple data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; S5: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

[0033] In the specific implementation process, firstly, a ternary data set including initial action data, scene data and text data is obtained, a bounding box is extracted for the scene data, and dimensions are filled for the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; Then, a scene encoder and a text encoder are constructed. The scene data and text data in the preprocessed ternary data set are encoded using the scene encoder and the text encoder respectively to obtain the scene features. and text features ; Then build a denoiser and randomly sample time steps , the action data in the preprocessed ternary data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features Input the denoiser together to get the denoised action data ; Then use the action data in the preprocessed triple data set And denoised motion data , jointly train the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; Finally, obtain new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ; By using bounding boxes to represent scenes, this method can capture scene information quickly and efficiently. At the same time, by fusing scene data and text descriptions, the generated actions not only meet the semantic requirements of text descriptions, but can also interact reasonably with specific scenes, effectively improving the stability and accuracy of generated actions.

[0034] Example 2 like Figure 2 As shown, this embodiment provides a multi-conditional human action generation method based on scenes and texts, comprising the following steps: S1: obtaining a ternary data set including initial action data, scene data and text data, performing bounding box extraction on the scene data, and performing dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; S2: Construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features and text features ; S3: Build denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; S4: Using the pre-processed action data in the triple data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; S5: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, is a positive integer; In step S1, the ternary data set includes a plurality of ternary data samples, each of which includes an initial action data, a scene data and a text data; The initial action data is specifically a frame sequence of key points of the human body; the scene data is specifically a point cloud of each object in the scene; the text data is specifically a text description of the action data and the scene data; In the step S1, initial motion data including several motions is obtained, and the data dimension of a single motion is (F, 24, 3), where F is the number of motion frames, each frame contains a static posture of the human body, and the static posture is represented by the three-dimensional coordinates of 24 human key points, and 3 indicates that the position of each human key point is represented by a three-dimensional space coordinate; the data dimensions of all motions are uniformly filled to a fixed number of frames using a zero-filling method. , and generate a mask at the same time, complete the dimension filling of the initial action data, and obtain the action data ; For the scene data in the ternary data set, the bounding boxes of each object in the scene are extracted; each of the bounding boxes represents an object, and the parameters of the i-th bounding box include shape information and category information ,in, Represents the center coordinates of the i-th bounding box; Indicates the size of the i-th bounding box; Represents the rotation angle of the i-th bounding box around the vertical axis; The scene data after bounding box extraction contains the bounding boxes of N objects. The zero-filling method is used to uniformly fill the scene data after bounding box extraction to a fixed number. , complete the dimension filling of the scene data after bounding box extraction; In step S2, the constructed scene encoder is based on the Transformer encoder structure, and the initial parameters are random parameters; For the bounding box of the i-th object , use two parallel multi-layer perceptrons to process the shape information of the i-th bounding box respectively and category information , get shape features and category features; concatenate the shape features and category features to get the i-th bounding box The eigenvector of ; For multiple bounding box features in scene data , using the Transformer encoder to perform scene-level global feature aggregation; the feature vector of each bounding box is aggregated through the self-attention mechanism in the Transformer encoder to capture the relationship between objects and global context information, and finally obtain the scene feature containing global information ; The text data is encoded using the text encoder in the pre-trained CLIP model to obtain the text features ; The step S3 comprises the following steps: Randomly sample time steps from a uniform distribution on the interval [0,T] , and according to the time step Action data Perform noise processing to generate noisy motion data ,in, Represents action data The i-th frame in Represents the action data after adding noise The i-th frame in; The formula for the noise addition process is as follows:

[0035] in, is standard Gaussian noise; is a predefined time step Decreasing noise weight function; The scene features and text features The dimensions are aligned through a linear layer respectively, and the two features after dimension alignment are added and fused to obtain the joint feature. , expressed as:

[0036] For the noise-added motion data Each frame in , using a multi-layer perceptron to process each frame As an action code , a total of action code units, represented by ; The time step is calculated using the following formula: Perform cosine position encoding:

[0037]

[0038] in, and Represents the time step Encoded values ​​on even and odd dimensions, is the dimension index, is the preset embedding dimension; The cosine position code is combined with the joint feature Add together to get the conditional code ; The denoiser is based on the Transformer encoder structure, and the initial parameters are random parameters; As the input of the denoiser, the output of the denoiser is the denoised codeword sequence , the action codewords in the denoised codeword sequence Through a linear layer, we get the denoised action data ; In step S4, the preset loss function is specifically a mean square error loss function, and the calculation formula is:

[0039] in, is the mean square error loss function value; is the total number of triple data samples in the triple data set; is the i-th action data ; is the denoised action data of the i-th item ; The step S5 comprises the following steps: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features ; The initial noise generated by random sampling , and new scene features and new text features input the trained denoiser together; The trained denoiser obtains the action data after the first round of iteration after the first round of denoising. ; Then for the time step Perform the noise addition operation to obtain the noise in the second round of iteration ; Then, the trained denoiser is used to perform a second round of denoising to obtain the action data after the second round of iteration ; Repeat the above process and use the trained denoiser to After a round of iteration, we finally get Action data after round iteration , and as the new action data , expressed as: .

[0040] In the specific implementation process, firstly, an "action-text-scene" ternary data set including initial action data, scene data and text data is obtained. In this embodiment, the data samples of the ternary data set are derived from the public data set HUMANISE (from the paper "HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes") and Trumans (from the paper "Scaling Up Dynamic Human-Scene Interaction Modeling"); wherein the action data is a frame sequence of key points of the human body, such as an action sequence of a person sitting on the bed; the scene data is a point cloud, such as a bed and other common objects that may exist; the text data is a simple description of the action and scene, such as "walk to the bedside" and "sit on the toilet"; After obtaining the ternary data set, it is necessary to perform preprocessing, extract the bounding box of the scene data, and fill the dimensions of the initial action data and the scene data after the bounding box extraction to obtain the preprocessed ternary data set; For the action data in the ternary data set, the dimension of a single action data is (F, 24, 3), where F represents the number of frames. Each frame contains a static posture of the human body, represented by the three-dimensional coordinates of 24 human key points. In the human skeleton motion capture data, "3" usually means that the position of each human key point is represented by a three-dimensional space coordinate, which is used to accurately locate the position of the human skeleton joint in the three-dimensional space. This is also a conventional representation method in the field of motion capture. Use zero filling method to fill each action data to a fixed number of frames The purpose of zero padding is to unify all action data to the same time step to ensure the consistency of data dimensions in subsequent model training and calculations. The padded data will be accompanied by a mask to identify which frames are generated by the padding operation. The mask will be used to exclude invalid padding parts in the subsequent loss function calculation to avoid interference with the model learning process. For the scene data in the triple data set, the existing open source target detection toolbox MMdetection3D is used to extract the bounding boxes of the objects in the scene; each bounding box represents an object in the scene and is defined by the following four types of information: center position ,size , around the vertical axis ( Axis) rotation angle and category labels ; After the extraction operation is completed, the single scene data in the triple data set is represented as N An unordered collection of bounding boxes; for scene data, zero padding is also used to uniformly fill each scene data to a fixed number. ; This padding method helps maintain the stability of the data structure and ensures that the model can handle action data of different lengths and scene data with different numbers of bounding boxes; Then, a scene encoder and a text encoder are constructed. The scene data and text data in the preprocessed ternary data set are encoded using the scene encoder and the text encoder respectively to obtain the scene features. and text features ; In this embodiment, the constructed scene encoder is based on the Transformer encoder structure, and the initial parameters are random parameters; For the bounding box of the i-th object , use two parallel multi-layer perceptrons to process the shape information of the i-th bounding box respectively and category information , get shape features and category features; concatenate the shape features and category features to get the i-th bounding box The eigenvector of ; For multiple bounding box features in scene data , using the Transformer encoder to perform scene-level global feature aggregation; the feature vector of each bounding box is aggregated through the self-attention mechanism in the Transformer encoder to capture the relationship between objects and global context information, and finally obtain the scene feature containing global information ; For the text encoder, this embodiment uses the text encoder in the public pre-trained CLIP model (its parameters are frozen and do not participate in subsequent training) to encode the text data to obtain text features ; like Figure 3 As shown, the denoiser is then constructed, with random sampling of time steps , the action data in the preprocessed ternary data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features Input the denoiser together to get the denoised action data ; Specifically, time steps are randomly sampled from a uniform distribution on the interval [0, T]. , and according to the time step Action data Perform noise processing to generate noisy motion data ,in, Represents action data The i-th frame in Represents the action data after adding noise The i-th frame in; The formula for noise addition is as follows:

[0041] in, is standard Gaussian noise, which is randomly sampled from a multivariate normal distribution with mean 0 and covariance matrix I; is a predefined time step The specific form of the decreasing noise weight function can be selected according to actual needs. Common noise weight functions are usually functions that decrease with time step t, such as exponential decay functions or linear decay functions. Selecting different decay functions can affect the noise control strategy in the generation process, thereby producing different effects on the generation results. In specific applications, the most appropriate noise attenuation function can be selected according to experimental results or task requirements; The scene features and text features The two features are dimensionally aligned through a linear layer to ensure that their dimensions are consistent; the two features after dimension alignment are added and fused to form a 512-dimensional joint feature. , expressed as:

[0042] For the noisy action data Each frame in , using a multi-layer perceptron to process each frame As an action code , each action code element feature dimension is 512 dimensions, a total of action code units, represented by ; Given a time step and embedding dimension =512, use the following formula to calculate the time step Perform cosine position encoding:

[0043]

[0044] in, and Represents the time step Encoded values ​​on even and odd dimensions, is the dimension index; Cosine position encoding and joint features Add together to get the conditional code ; The denoiser is based on the Transformer encoder structure, and the initial parameters are random parameters. As the input of the denoiser, the output of the denoiser is the denoised codeword sequence , the action codewords in the denoised codeword sequence Through a linear layer, we get the denoised action data ; Then use the action data in the preprocessed triple data set And denoised motion data , using the mean square error loss function to jointly train the scene encoder and the denoiser to obtain a trained scene encoder and a trained denoiser; In the training process, this embodiment uses the mean square error as the loss function to measure the difference between the action data output by the denoiser and the actual action data without noise. The loss function calculation formula is as follows:

[0045] in, is the mean square error loss function value; is the total number of triple data samples in the triple data set; is the i-th action data ; is the denoised action data of the i-th item ; Finally, obtain new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and text encoder respectively. and new text features ; Set the time step , random sampling generates initial noise , time step It is a fixed value, and common values ​​are 50, 500, and 1000; The initial noise generated by random sampling , and new scene features and new text features Jointly input the trained denoiser; The trained denoiser gets the action data after the first round of denoising. ; Then for the time step Perform the noise addition operation to obtain the noise in the second round of iteration ; Then use the trained denoiser to perform the second round of denoising to obtain the action data after the second round of iteration ; Repeat the above process and use the trained denoiser to After a round of iteration, we finally get Action data after round iteration , and as the new action data , expressed as: ; By using bounding boxes to represent scenes, this method can capture scene information quickly and efficiently. At the same time, by fusing scene data and text descriptions, the generated actions not only meet the semantic requirements of text descriptions, but can also interact reasonably with specific scenes, effectively improving the stability and accuracy of generated actions.

[0046] Example 3 like Figure 4 As shown, this embodiment provides a multi-conditional human motion generation system based on scenes and texts, which applies the multi-conditional human motion generation method based on scenes and texts, including: The data preprocessing unit 301 is used to obtain a ternary data set including initial action data, scene data and text data, perform bounding box extraction on the scene data, and perform dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; Feature extraction unit 302: used to construct a scene encoder and a text encoder, and respectively encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder to obtain scene features. and text features ; Prediction result generation unit 303: used to build a denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; Joint training unit 304: for using the action data in the preprocessed ternary data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; Action generation unit 305: used to obtain new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

[0047] In the specific implementation process, first, the data preprocessing unit 301 obtains a ternary data set including initial action data, scene data and text data, performs bounding box extraction on the scene data, and performs dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; Then the feature extraction unit 302 constructs a scene encoder and a text encoder, and encodes the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain the scene feature and text features ; Then the prediction result generation unit 303 constructs a denoiser, randomly sampling time steps , the action data in the preprocessed ternary data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features Input the denoiser together to get the denoised action data ; Then the joint training unit 304 uses the action data in the preprocessed ternary data set And denoised motion data , jointly train the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; Finally, the action generation unit 305 obtains new scene data and performs bounding box extraction, as well as new text data, and generates new scene features through the trained scene encoder and text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ; By using bounding boxes to represent scenes, the system can capture scene information quickly and efficiently. At the same time, by fusing scene data and text descriptions, the generated actions not only meet the semantic requirements of text descriptions, but can also interact reasonably with specific scenes, effectively improving the stability and accuracy of generated actions.

[0048] The same or similar reference numerals correspond to the same or similar components; The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present invention; Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-conditional human action generation method based on scenes and texts, characterized in that: The following steps are involved: S1: obtaining a ternary data set including initial action data, scene data and text data, performing bounding box extraction on the scene data, and performing dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; S2: Construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features and text features ; S3: Build denoiser, randomly sample time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added action data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; S4: Using the pre-processed action data in the triple data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; S5: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

2. The method for generating multi-conditional human motion based on scenes and texts according to claim 1, characterized in that: In step S1, the ternary data set includes a plurality of ternary data samples, each of which includes an initial action data, a scene data and a text data; The initial action data is specifically a frame sequence of key points of the human body; the scene data is specifically a point cloud of each object in the scene; and the text data is specifically a text description of the action data and the scene data.

3. The method for generating multi-conditional human motion based on scenes and texts according to claim 1, characterized in that: In the step S1, initial motion data including several motions is obtained, and the data dimension of a single motion is (F, 24, 3), where F is the number of motion frames, each frame contains a static posture of the human body, and the static posture is represented by the three-dimensional coordinates of 24 human key points, and 3 indicates that the position of each human key point is represented by a three-dimensional space coordinate; the data dimensions of all motions are uniformly filled to a fixed number of frames using a zero-filling method. , and generate a mask at the same time, complete the dimension filling of the initial action data, and obtain the action data ; For the scene data in the ternary data set, the bounding boxes of each object in the scene are extracted; each of the bounding boxes represents an object, and the parameters of the i-th bounding box include shape information and category information ,in, Represents the center coordinates of the i-th bounding box; Indicates the size of the i-th bounding box; Represents the rotation angle of the i-th bounding box around the vertical axis; The scene data after bounding box extraction contains the bounding boxes of N objects. The zero-filling method is used to uniformly fill the scene data after bounding box extraction to a fixed number. , complete the dimension filling of the scene data after the bounding box is extracted.

4. The method for generating multi-conditional human motion based on scenes and texts according to claim 3, characterized in that: In step S2, the constructed scene encoder is based on the Transformer encoder structure, and the initial parameters are random parameters; For the bounding box of the i-th object , use two parallel multi-layer perceptrons to process the shape information of the i-th bounding box respectively and category information , get shape features and category features; concatenate the shape features and category features to get the i-th bounding box The eigenvector of ; For multiple bounding box features in scene data , using the Transformer encoder to perform global feature aggregation at the scene level; the feature vector of each bounding box is aggregated through the self-attention mechanism in the Transformer encoder to capture the relationship between objects and global context information, and finally obtain the scene feature containing global information ; The text data is encoded using the text encoder in the pre-trained CLIP model to obtain the text features .

5. The method for generating multi-conditional human motion based on scenes and texts according to claim 1, characterized in that: The step S3 comprises the following steps: Randomly sample time steps from a uniform distribution on the interval [0,T] , and according to the time step Action data Perform noise processing to generate noisy motion data ,in, Represents action data The i-th frame in Represents the action data after adding noise The i-th frame in ; The formula for the noise addition process is as follows: in, is standard Gaussian noise; is a predefined time step Decreasing noise weight function; The scene features and text features The dimensions are aligned through a linear layer respectively, and the two features after dimension alignment are added and fused to obtain the joint feature. , expressed as: For the noise-added motion data Each frame in , using a multi-layer perceptron to process each frame As an action code , a total of action code units, represented by ; The time step is calculated using the following formula: Perform cosine position encoding: in, and Represents the time step Encoded values ​​on even and odd dimensions, is the dimension index, is the preset embedding dimension; The cosine position code is combined with the joint feature Add together to get the conditional code ; The denoiser is based on the Transformer encoder structure, and the initial parameters are random parameters; As the input of the denoiser, the output of the denoiser is the denoised codeword sequence , the action codewords in the denoised codeword sequence Through a linear layer, we get the denoised action data .

6. The method for generating multi-conditional human motion based on scenes and texts according to claim 1, characterized in that: In step S4, the preset loss function is specifically a mean square error loss function, and the calculation formula is: in, is the mean square error loss function value; is the total number of triple data samples in the triple data set; is the i-th action data ; is the denoised action data of the i-th item .

7. The method for generating multi-conditional human motion based on scenes and texts according to claim 1, characterized in that: The step S5 comprises the following steps: Acquire new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively. and new text features ; The initial noise generated by random sampling , and new scene features and new text features input the trained denoiser together; The trained denoiser obtains the action data after the first round of iteration after the first round of denoising. ; Then for the time step Perform the noise addition operation to obtain the noise in the second round of iteration ; Then, the trained denoiser is used to perform a second round of denoising to obtain the action data after the second round of iteration ; Repeat the above process and use the trained denoiser to After a round of iteration, we finally get Action data after round iteration , and as the new action data , expressed as: .

8. A scene and text-based multi-conditional human motion generation system, using the scene and text-based multi-conditional human motion generation method described in any one of claims 1 to 7, characterized in that: include: Data preprocessing unit: used to obtain a ternary data set including initial action data, scene data and text data, perform bounding box extraction on the scene data, and perform dimension filling on the initial action data and the scene data after bounding box extraction to obtain a preprocessed ternary data set; Feature extraction unit: used to construct a scene encoder and a text encoder, and encode the scene data and text data in the preprocessed ternary data set using the scene encoder and the text encoder respectively to obtain scene features. and text features ; Prediction result generation unit: used to build a denoiser, randomly sampling time steps , the action data in the preprocessed triple data set Perform noise processing to obtain the noised motion data ; The noise-added motion data , Scene Features and text features The denoiser is inputted together to obtain the denoised action data ; Joint training unit: used to utilize the action data in the preprocessed ternary data set And denoised motion data , jointly training the scene encoder and the denoiser using a preset loss function to obtain a trained scene encoder and a trained denoiser; Action generation unit: used to obtain new scene data and perform bounding box extraction, as well as new text data, and generate new scene features through the trained scene encoder and the text encoder respectively and new text features , generate initial noise by random sampling , using the trained denoiser The denoising process is iterated step by step, and finally new action data is generated ,in, Is a positive integer.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.

Images