Method and device for generating human motion through fine-grained text, medium and product

Abstract

The invention discloses a method, a device, a medium and a product for generating human motion through a fine-grained text, and relates to the technical field of computer vision. The method comprises the following steps: acquiring related information of to-be-generated human motion; the related information comprises operation information, fine-grained text condition information and space control signal information; encoding the related information to obtain features of the related information; denoising the features of the related information to obtain potential motion features; decoding the motion potential features to obtain an operation sequence; and performing post-processing on the operation sequence to obtain operation information of the to-be-generated human motion. According to the invention, accurate control and generation of human body motion are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, medium and product for generating human body motion from fine-grained text. Background Art

[0002] In the field of computer vision, generating human motion that meets specific conditions has always been a challenging task. In recent years, with the development of diffusion models, text-based conditional human motion generation has made significant progress in generating efficient human motion. However, it is still challenging to generate motion outside the distribution of the original dataset, i.e., zero-sample generation. Secondly, existing methods generally lack fine-grained controllability of motion, which makes it difficult to meet the needs of complex scenes in practical applications. In particular, during the generation process, it is still challenging to require the generated motion to meet the user's expectations, which greatly limits the applicability of these methods.

[0003] Although incorporating spatial constraints is an essential feature, it remains a challenge for text-based human motion synthesis methods. An ideal method should ensure that the generated motion closely follows the global spatial control signal, aligns with the text semantics, and maintains fidelity. Such an approach should also be able to control any joint and its combination, as well as handle sparse control signals. Traditional methods are limited by their relative human pose representation, where the positions of other joints are defined relative to the pelvis. These methods have difficulty incorporating global constraints on joints other than the pelvis and handling sparse spatial constraints. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, medium and product for generating human motion from fine-grained text, which solves the problem of insufficient fine-grained controllability of human motion generation in existing methods, and realizes precise control and generation of human motion by innovatively combining motion information, fine-grained text condition information and spatial control signal information.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for generating human body motion from fine-grained text, the method comprising:

[0007] Obtaining relevant information of the human body movement to be generated; the relevant information includes operation information, fine-grained text condition information and spatial control signal information;

[0008] Encoding the relevant information to obtain features of the relevant information;

[0009] De-noising the features of the relevant information to obtain motion potential features;

[0010] Decoding the motion potential features to obtain a running sequence;

[0011] The operation sequence is post-processed to obtain operation information of the human body movement to be generated.

[0012] Optionally, a pre-trained CLIP text encoder is applied to encode the fine-grained text condition information.

[0013] Optionally, a pre-trained multi-layer neural network is used to encode the spatial control signal information.

[0014] Optionally, a diffusion model is applied to denoise the features of the relevant information; the denoising process of the diffusion model includes a forward denoising process and a reverse denoising process.

[0015] Optionally, the forward noisy process follows a Markov chain.

[0016] Optionally, the Markov chain is:

[0017]

[0018] Among them, z n is the potential noise feature in the process of adding noise in the nth step; z n-1 is the potential noise feature in the process of adding noise in the n-1th step; α n It is a hyperparameter used to adjust the degree of noise addition; I is the unit matrix; is the normal distribution; q is the conditional probability distribution.

[0019] A device for generating human body motion from fine-grained text, the device comprising:

[0020] An input module, used to obtain relevant information of the human body movement to be generated; the relevant information includes operation information, fine-grained text condition information and space control signal information;

[0021] An encoding module, used to encode the relevant information to obtain features of the relevant information;

[0022] A denoising module, used for denoising the features of the relevant information to obtain motion potential features;

[0023] A decoding module, used for decoding the motion potential features to obtain a running sequence;

[0024] The post-processing module is used to post-process the operation sequence to obtain the operation information of the human body movement to be generated.

[0025] An electronic device, comprising:

[0026] one or more processors;

[0027] a memory for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-mentioned methods for generating human body motion from fine-grained text.

[0030] A computer program product comprises a computer program, which, when executed by a processor, implements any of the above-mentioned methods for generating human body motion from fine-grained text.

[0031] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0032] The present invention discloses a method, device, medium and product for generating human motion from fine-grained text. By acquiring motion information of input content motion, fine-grained text condition information and spatial control signal information, motion content features, fine-grained text features and spatial control signal features are obtained by encoding, and the above three features are used as conditions to guide the denoising process of Gaussian noise in the motion potential space. The process is implemented by a multi-condition denoising network, and finally a controllable motion potential feature is obtained, and the feature is restored by a motion decoder to obtain human motion that meets the text conditions and spatial control signals. The present invention realizes the generation of human motion that meets the requirements of spatial control signals and fine-grained text conditions by encoding content motion, text conditions and spatial control signals into features as conditional guidance motion generation process, introduces flexible spatial control signals in the text-to-motion generation process, and allows multiple joints to be controlled at different time points, so that the generated human motion is more realistic, coherent, and meets specific spatial constraints. This method is of great significance for improving the accuracy and realism of human motion generation and expanding its application in different complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic flow chart of a method for generating human body motion from fine-grained text provided in Example 1 of the present invention;

[0035] Figure 2 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The purpose of the present invention is to provide a method, device, medium and product for generating human motion from fine-grained text, aiming to solve the problem of insufficient fine-grained controllability of human motion generation in existing methods, and to achieve precise control and generation of human motion by innovatively combining motion information, fine-grained text condition information and spatial control signal information.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, the method for generating human body motion from fine-grained text in this embodiment includes:

[0041] Step S1: Obtain relevant information of the human body motion to be generated; the relevant information includes motion information, fine-grained text condition information (including global text and local text, and local text includes text of arms, legs, torso and neck) and spatial control signal information.

[0042] Step S2: Encode the relevant information to obtain features of the relevant information; the features of the relevant information include motion features, text features, and space control signal features.

[0043] Specifically, a pre-trained CLIP text encoder is used to encode the fine-grained text condition information. A pre-trained multi-layer neural network is used to encode the spatial control signal information.

[0044] In practical applications, input motion information, fine-grained text condition information, and spatial control signal information are obtained.

[0045] Among them, the motion information is encoded through the motion coding network to obtain motion features. The motion coding network is used to extract dynamic features from the input motion information and provide a detailed understanding of the motion pattern. The motion coding network uses the Transformer network; the motion data is input as a sequence in time steps and is first processed by linear embedding to meet the input requirements of the Transformer. Through the multi-layer Transformer encoder, the motion sequence information is encoded to extract the global motion pattern. The attention mechanism (self-attention) of the Transformer encoder helps to model the complex relationships between different joints of the human body.

[0046] The fine-grained text conditional information is encoded through a text encoding network to obtain text features. First, a large language model (such as ChatGPT) is used to parse the input text into fine-grained information describing different body parts, and then these fine-grained text descriptions are encoded through multiple pre-trained CLIP text encoders (global descriptions and local descriptions are encoded using CLIP texts respectively), and then the global text features and local text features are fused to generate overall text features.

[0047] The spatial control signal information is encoded through a spatial control coding network to obtain spatial control signal features. The spatial control signal defines the spatial constraints when generating motion, allowing the user to specify any joint or part in the motion for precise control. The spatial control signal coding network includes a multi-layer neural network for extracting spatial features from the input spatial control signal. For encoding complex spatial control signals, especially when involving control of multiple joints, the following neural networks are recommended: (1) Convolutional neural network (CNN): suitable for extracting local spatial features. (2) Graph convolutional neural network (GCN): suitable for processing complex relationships between joints. (3) Transformer network: suitable for capturing global dependencies between joints, especially in complex control scenarios.

[0048] The text encoding network includes a multi-layer neural network for extracting semantic features from input fine-grained text. The text encoding network includes a pre-trained CLIP text encoder for encoding fine-grained text condition information into text features. The motion decoder is pre-trained in conjunction with the motion encoding network to efficiently restore motion features in the latent space to actual human motion sequences.

[0049] Step S3: De-noising the features of the relevant information to obtain motion potential features.

[0050] Specifically, a diffusion model is applied to denoise the features of the relevant information; the denoising process of the diffusion model includes a forward denoising process and a reverse denoising process.

[0051] Furthermore, the forward noise addition process follows a Markov chain. The Markov chain is:

[0052]

[0053] Among them, z n is the potential noise feature in the process of adding noise in the nth step; z n-1 is the potential noise feature in the process of adding noise in the n-1th step; α n It is a hyperparameter used to adjust the degree of noise addition; I is the unit matrix; is the normal distribution; q is the conditional probability distribution.

[0054] In practical applications, the motion features, text features, and spatial control signal features obtained in step S1 are input as conditions into the latent space of the diffusion model to guide the denoising process in the diffusion model. During the diffusion process, the multi-conditional denoising network removes noise based on the input motion features, text features, and spatial control signal features to generate motion latent features that meet the requirements.

[0055] In the diffusion model, "denoising" is the core process of generating data, which gradually recovers the target data (here, the human action sequence) from the noise. The multi-conditional denoising network means that in the denoising process, the network will use multiple conditional inputs to guide the generation of each step, so that the generated results meet specific requirements. The following are several key features of the multi-conditional denoising network: (1) Multi-conditional input: motion features, text features, and spatial control signal features are used as conditional inputs, providing different guidance through different feature modalities. (2) Motion features: Features from previous action sequences help generate continuity consistent with the context action. (3) Text features: Description of the action, such as "raise your left hand" or "bend forward", these features determine what the target of the generated action is. (4) Spatial control signal features: Control the specific execution position or spatial relationship of the action, for example, the position of the hand should be close to an object. These signals are used to generate actions that meet the requirements of a specific scene. (5) Network structure: The multi-conditional denoising network combines these conditional features in each denoising step, making the model generation process more controllable and able to finely control the position and action state of each joint. (6) Multiple conditions refer to the use of feature information from multiple sources in the process of generating the model to ensure that the generated human body movements meet the expected multiple requirements. These conditions include: 1) Motion features: reflecting the dynamic characteristics of the current generated movement. 2) Text features: providing a semantic description of the generated movement. 3) Spatial control signal features: used to constrain the spatial position of the generated movement, such as controlling the specific position of a joint or the overall trajectory.

[0056] Motion features, text features, and spatial control signal features are input into the diffusion model as conditions, and participate in the generation process through feature fusion, guiding the model to generate motion that meets these conditions during the step-by-step denoising process. The latent space is a high-dimensional space that represents data inside the diffusion model. The model generates motion data that meets the conditions from random noise by performing step-by-step denoising in this space.

[0057] Step S4: Decode the motion potential features to obtain a running sequence.

[0058] In practical applications, the generated motion latent features are input into the motion decoder, and the features are restored to a human motion sequence that meets the text conditions and spatial control signal requirements. Preferably, the motion decoder is a Transformer decoder. A human motion sequence refers to a serialized representation of the continuous motion information of a human body over a period of time. Specifically, it includes the following: (1) Joint position sequence: The position of each joint of the human body at different times, usually represented by three-dimensional coordinates. For example, the coordinate values ​​of key joints such as the wrist, elbow, and knee at different time steps. (2) Posture and posture information: Not only the position of the joints, but also the angles between the joints, rotational postures and other information may be included to fully describe the posture changes of the human body. (3) Time step: The human motion sequence is represented by time steps, so it can describe the entire process of an action. For example, an action (such as waving) will contain multiple time steps from the starting posture to the ending posture, and each time step corresponds to a complete joint state.

[0059] The motion decoder is pre-trained together with the motion encoding network to efficiently restore the motion features in the latent space to a motion sequence that meets the requirements. The generated motion sequence not only meets the input fine-grained text condition information, but also achieves flexible control of the motion through the guidance of the spatial control signal, and can generate human motion beyond the scope of the original dataset in different scenarios.

[0060] Step S5: Post-processing the running sequence to obtain running information of the human body movement to be generated. The post-processing operation includes a motion smoothing algorithm and a posture correction algorithm to ensure that the generated motion sequence has high visual authenticity and naturalness.

[0061] In practical applications, the generated motion sequence is post-processed to ensure its consistency and coherence in time and space. Specifically, after the generated motion sequence is post-processed, the signal obtained is a corrected high-quality human motion signal, which means that the generated motion sequence can have better consistency and coherence in time and space after post-processing. Specifically, the purpose of post-processing is to enhance the visual and physical realism of the generated motion, making the generated motion more natural and in line with physical laws.

[0062] Through steps S1 to S5, the present invention realizes fine-grained control of human motion, can generate high-quality human motion sequences that meet input text conditions and spatial control signals, and is suitable for virtual reality, animation production and other fields.

[0063] The present invention adopts a divide-and-conquer strategy and proposes a new framework, called a controllable fine-grained human motion diffusion model, for zero-shot human motion generation. Specifically, the previous fuzzy text annotations are first parsed into fine-grained descriptions of different body parts by using a large language model. Then these fine-grained descriptions are used to guide the Transformer-based diffusion model, which further adopts a partial labeling design. The partial labeling design is mainly reflected in the selective control of conditions such as motion features, text features, and spatial control signals, so that the model can flexibly control specific joints or motion parts according to user needs. At the same time, spatial control signals are added, and users can specify spatial control signals to control any joint in the motion, thereby generating human motion that meets their expectations. Since the text is closer to the description of the essence of the motion, human motion that exceeds the scope of the original data set can be generated and meets the user's expectations. Spatial control signals refer to control information used to constrain joint positions or motion trajectories in the process of generating human motion. These signals can be used to accurately describe the target position or trajectory of a joint in space, ensuring that the generated motion meets specific spatial requirements. Spatial control signals correspond to specific joints in human motion, and are used to control the positions of these joints during the generation process. For example, the user can specify the height range of the knee in a certain action, and the model will refer to this control signal when generating the action to adjust the movement of the knee. Therefore, the spatial control signal is used to constrain the movement of the human body in space, including the position and trajectory of the joints. It is directly related to the relationship between the human body movement. These control signals are used to ensure that the generated action meets the specific space requirements and user expectations.

[0064] The present invention improves the text dataset part of the original human motion dataset HumanML3D. The original text only describes a complete human motion information. On this basis, a new action description of the most important human body parts in the motion is added, and this is used as a new text dataset for training and testing.

[0065] First, the present invention forms complete input data by acquiring motion information, fine-grained text condition information and spatial control signal information of the input content. The motion information describes the dynamic characteristics in the input content, the fine-grained text condition information provides semantic instructions for generating motion, and the spatial control signal information is used to define the spatial constraints when generating motion.

[0066] Then, the input motion information, fine-grained text condition information and spatial control signal information are respectively encoded to obtain motion content features, fine-grained text features and spatial control signal features. After encoding, these features can provide accurate and guiding condition information for the subsequent generation process.

[0067] In the generation phase, the above three features are used as conditions and combined with the denoising process of Gaussian noise in the motion latent space. The core of this denoising process is to remove random noise that may affect the quality of motion generation while retaining the motion and semantic constraints provided by the input features. Through this process, highly controllable motion latent features can be generated, which not only meet the input text conditions but also meet the requirements of spatial control signals.

[0068] Finally, the generated motion latent features are decoded by the motion decoder, and the actual human motion that meets the text conditions and the requirements of the spatial control signal is finally obtained. This motion can not only accurately reflect the input semantic information, but also has a high degree of spatial constraint and realism.

[0069] The present invention can be applied to a virtual fitness coach system. In the virtual fitness coach system, users need to follow the movements of the virtual coach to exercise. In order to improve the training effect and experience of the user, the movements of the virtual coach need to be accurate and consistent with the instructions, and can be dynamically generated according to the specific needs of the user. The virtual fitness coach can generate accurate fitness movements according to the user's instructions, and can be adjusted according to different constraints in the scene (such as spatial control signals) (when completing a certain action, the controlled body part must coincide with the spatial control signal and follow the constraints of the spatial control signal, and the generation of the spatial control signal is determined by the user). These generated movements meet the standards of fitness and the expectations of the user, and have a high degree of visual authenticity and coherence.

[0070] The advantage of the present invention is that by encoding content motion, fine-grained text conditions and spatial control signals into features and guiding them as conditions in the motion generation process, the problem of insufficient fine-grained control in the prior art is solved. Compared with the traditional method, the present invention can not only generate accurate motion that meets the text conditions, but also achieve flexible control of the motion by introducing spatial control signals.

[0071] Compared with the prior art, the present invention obtains motion features, text features and spatial control signal features by respectively passing the input motion information, fine-grained text condition information and spatial control signal information through the motion coding network, text coding network and spatial control coding network. The motion coding network can effectively extract the features of the input motion, the text coding network uses the pre-trained CLIP encoder to accurately capture the text condition information, and the spatial control coding network converts the spatial control signal into a spatial feature that can be used by the diffusion model. In the present invention, the motion features, text features and spatial control signal features obtained above are input as conditions into the diffusion model of the latent space to guide its denoising process. The denoising process is implemented by a multi-condition denoising network, which can accurately generate motion latent features that meet the text conditions and spatial control requirements. Then, these motion latent features are decoded by a motion decoder and restored to actual human motion sequences. The method of the present invention can flexibly generate fine-grained human motion that meets multiple conditions by integrating motion information, text conditions and spatial control signals. In particular, by introducing the spatial control signal, the controllability and scene adaptability of the motion are greatly enhanced, and the common motion generation incoherence problem in traditional methods is avoided. This makes the generated motion more in line with user expectations and meet user needs while being natural, coherent and in line with text conditions.

[0072] Example 2

[0073] Based on Example 1, the present invention also provides a system for generating human motion from fine-grained text, including: an input module, an encoding module, a denoising module, a decoding module and a post-processing module.

[0074] The input module is used to obtain relevant information of the human body movement to be generated; the relevant information includes operation information, fine-grained text condition information and space control signal information.

[0075] The encoding module is used to encode the relevant information to obtain the characteristics of the relevant information.

[0076] Specifically, the encoding module is connected to the input module; the encoding module is used to encode the motion information, fine-grained text condition information and space control signal information into motion features, text features and space features respectively.

[0077] The denoising module is used to denoise the features of the relevant information to obtain motion potential features.

[0078] Specifically, the denoising module is connected to the encoding module; the denoising module is used to perform a denoising process under the guidance of motion features, text features and spatial features to generate motion potential features that meet the conditions.

[0079] The decoding module is used to decode the motion potential features to obtain a running sequence.

[0080] Specifically, the decoding module is connected to the denoising module; the decoding module is used to decode the motion potential features into actual human motion sequences.

[0081] The post-processing module is used to post-process the operation sequence to obtain the operation information of the human body movement to be generated.

[0082] Specifically, the post-processing module is connected to the decoding module; the post-processing module is used to post-process the generated motion sequence to ensure its coherence and realism.

[0083] The following takes a virtual fitness coach system as an example to introduce the application scenario of the present invention.

[0084] The virtual fitness coach system uses computer devices to realize motion generation based on fine-grained text descriptions, helping users get targeted fitness guidance. The system includes user input terminals (such as tablets, mobile phone applications), virtual coach display devices (such as TV screens, VR headsets), cameras and sensors, and a central computer device as the core controller of the system. The system helps users understand and perform correct fitness postures by generating standardized fitness movements that meet user needs.

[0085] In the virtual coaching system, the detailed application steps of the present invention are as follows:

[0086] Step 1: The user inputs fitness action requirements.

[0087] (1) User interface interaction.

[0088] The user enters specific fitness action requirements in the input terminal (such as a mobile phone or tablet application), describing the type and details of the action. The text includes global text and local text. The global text describes a specific action, such as doing push-ups, squats, etc., and the local text is the action of the arms, legs, torso and neck when completing the exercise. The user can also enter personal physical conditions (such as height, weight, physical fitness level, etc.) to help the system generate fitness actions suitable for their physical conditions. The input text description is transmitted to the input / output interface of the central computer device through the network.

[0089] Among them, the fitness action requirements input by the user are divided into global text and local text, which describe the overall action and the action of specific body parts respectively.

[0090] 1. Global text description: T global ={t1,t2,…,tn}.

[0091] Among them, Tglobal Represents the global action description of the user input, t i Represents a word or phrase in the global text.

[0092] 2. Local text description: Local text is used to describe the action of a specific body part, including descriptions of arms, legs, torso, and neck: local ={T arms ,T legs ,T torso ,T neck}.

[0093] Among them: arms text: T arms ={t arms ,1,t arms,2 ,…,t arms,n}; T arms Indicates a partial description of the arm's movement, such as "keep your arms extended."

[0094] Legs Text: T legs ={t legs,1, t legs,2 ,…,t legs,n}. T legs Indicates a partial description of the leg movement, such as "Raise your left leg to 90 degrees and hold for 5 seconds."

[0095] torso (torso) text: Ttorso = {t torso,1, t torso,2 ,…,t torso,n}; T torso Indicates a partial description of the torso's movements, such as "keep your upper body straight."

[0096] neck text: T neck ={t neck,1 ,t neck,2 ,…,t neck,n}; T neck Indicates a local description of the neck action, such as "keep the neck straight."

[0097] The user's physical condition is represented as a vector: S u =[h,w,f]; where h represents the user's height, w represents the weight, and f represents the user's physical fitness level.

[0098] (2) User data collection.

[0099] The system collects information such as the user's current posture and spatial position through sensors and cameras. For example, whether the user is in an open space and the current standing posture. This information is transmitted to the computer device to guide the generation of fitness movements that are suitable for the current environment and physical condition.

[0100] Step 2: Feature encoding.

[0101] (1) Text feature encoding.

[0102] The fitness action requirements input by the user are divided into global text and local text, which describe the overall action and the action of a specific body part respectively. For example: do push-ups. Bend your arms to support your body, straighten them when pushing up, and keep them shoulder-width apart. Straighten your legs together, with your toes on the ground, and keep your body stable. Keep your torso straight, forming a straight line from head to toe, without sagging your waist or arching your back. Keep your neck naturally extended, look slightly forward, and avoid lowering or raising your head. The processor of the computer device calls the text encoding module to encode the fitness description text input by the user. The text encoding module converts the text into a feature vector that describes the specific action intention.

[0103] Global text feature encoding: The input global text T is encoded by a text encoder (such as CLIP). global Encoded as text feature vector F global : F global =f text (T global ), where f text Represents the text encoding function, F global A feature vector representing the global context.

[0104] Local text feature encoding: The features of each local text are encoded as feature vectors:

[0105] F arms =f text (T arms );

[0106] F legs =f text (T legs );

[0107] F torso =ftext(T torso );

[0108] F neck =f text (T neck );

[0109] Among them, F arms 、F legs 、F torso 、Fneck Feature vectors representing the local text of the arms, legs, torso, and neck, respectively.

[0110] (2)Encoding of environmental and physical characteristics.

[0111] The processor encodes the user's current posture and spatial position obtained from the camera and sensor through the spatial control feature encoding module.

[0112] Environmental feature encoding: describes the constraints in the space around the user, such as whether the distance from the surrounding furniture is sufficient to lift the legs.

[0113] Body feature coding: Generate body feature coding based on the height, weight and other information entered by the user. For example, the angle of leg raising may be different for users of different body shapes. This information helps the system generate actions suitable for individual users.

[0114] Environmental feature encoding: The user's environmental features are obtained through sensor data, expressed as:

[0115] F env =f env (E);

[0116] Among them, E represents environmental information, F env Represents the environmental feature vector.

[0117] Physical feature encoding: Encoding the user's physical condition S u Encoded as a body feature vector:

[0118] F S =f body (S u );

[0119] Among them, f body The encoding function representing the user's physical characteristics, F S Represents the body feature vector.

[0120] Step 3: Feature vector fusion.

[0121] The global features, local features, environmental features and body features are fused to obtain a comprehensive feature vector to guide motion generation.

[0122] Fusion of local feature vectors: All local features are fused into one feature vector: F local =F arms +F legs +F torso +F neck Among them, F local A comprehensive feature vector representing local features.

[0123] Fusion of global features and local features: The global feature vector F global and the local eigenvector F local To fuse: F text =F global +F local Among them, F text A comprehensive feature vector representing the text description.

[0124] Fusion of the final feature vector: Fusion of the text feature vector, environment feature vector and body feature vector into a comprehensive feature vector:

[0125] F combined =F text +F env +F S ;

[0126] Among them, F combined It is the fused action feature vector, which contains global and local action requirements, environmental features and body features.

[0127] Step 4: The diffusion model generates motion latent features, including the diffusion model denoising and denoising process.

[0128] The text features, environmental features, and physical features are input into a diffusion model stored in a computer device.

[0129] Positive Noise: Simulates the motion deviations that may be encountered during fitness (for example, the small shaking that occurs during leg lifting). The noise addition process increases the diversity of motion generation.

[0130] Reverse denoising: Combined with the user's specific needs (such as the time to maintain balance and the height of leg lifting), the model generates motion latent features that meet the requirements through a gradual denoising process.

[0131] In the forward noise stage, random noise is added to the features:

[0132] F noisy =F combined +∈,∈~N(0,σ 2 );

[0133] Among them, ∈ represents random noise that follows a normal distribution, σ 2 represents the noise standard deviation.

[0134] In the reverse denoising stage, the noise is gradually removed through the diffusion model to generate the latent motion features: F latent =f denoise (F noisy ), where f denoise represents the diffusion model denoising process, F latent Represents the generated motion latent features.

[0135] Step 5: Decode and generate standardized fitness movements of the virtual coach, specifically: call the decoding module to generate an action sequence.

[0136] The processor calls the decoding module to decode the potential motion features into actual fitness action sequences.

[0137] Generate specific actions: The action includes the entire process from standing to gradually raising the left leg. The sequence generated by the decoding module includes the joint position and action posture of the virtual coach in each frame. Including how to maintain balance, the specific angle of leg raising, and the fine-tuning process of the body's center of gravity.

[0138] The decoding module generates action sequences: The decoder converts the motion latent features F latent Transformed into action sequence M: M = f decode (F latent ), where f decode represents the decoding function, M={m1,m2,…,m t} represents the action sequence, m i represents the joint position and pose at time step i.

[0139] Step 6: Post-processing and motion quality optimization, including smoothing and posture correction.

[0140] (1) Smoothing. The decoded action sequence may have some abrupt changes. The processor calls the post-processing module to smooth these actions to ensure that the virtual coach's movements are visually natural and smooth. Smoothing ensures that there is no jitter or break in the entire process from raising the leg to maintaining balance and slowly lowering it.

[0141] Smoothing: Smoothing the generated action sequence M to remove mutations and incoherence: M smooth =f smooth (M); where f smooth represents a smooth function, M smooth Represents the action sequence after smoothing.

[0142] (2) Posture correction. The processor uses the posture correction module to ensure that the virtual coach's movements are ergonomic. For example, it ensures that the body does not tilt when raising the left leg and the center of gravity is within a reasonable range, so that the coach's movements can provide the user with a correct reference.

[0143] Posture correction: adjust the posture to be ergonomic and generate the final standardized action sequence: M final =f correct ; Among them, f correct represents the posture correction function, M final Represents the final sequence of actions.

[0144] Step 7: Action demonstration and user guidance.

[0145] (1) Virtual coach action display. The generated fitness action is presented on a display device (such as a TV screen or VR device) by a virtual coach. Specifically, the action sequence M is displayed on the virtual coach display device. final , users train according to the virtual coach.

[0146] Users can watch the fitness movements demonstrated by the virtual coach through a large screen or VR headset. The virtual coach's movements are presented synchronously, including the angle of leg raising, holding time and lowering speed, ensuring that users can clearly understand how to perform the entire movement.

[0147] (2) Camera monitoring and user feedback. The system monitors the user's movements in real time through the camera and compares them with the standard movements of the virtual coach. For example, the camera can detect whether the user's leg is raised to 90 degrees and whether the balance time meets the requirements.

[0148] The system captures the user's actual actions through the camera, expressed as: M user =f camera (C); where C represents the data captured by the camera, M user Represents the sequence of actions actually performed by the user.

[0149] (3) Movement evaluation. The system evaluates the user’s movements in real time and provides feedback on the screen, such as “Your balance time is less than 5 seconds, please keep working on it” or “Your leg is not raised high enough, please raise it higher.” This feedback helps users continuously optimize their movements during practice.

[0150] Action evaluation: The action sequence M performed by the user user With standard action sequence M final Compare, evaluate deviations and generate feedback F feedback : F feedback =f compare (M user ,M final ), where f compare Represents a comparison and evaluation function.

[0151] Step 8: User feedback and personalized adjustments.

[0152] (1) Personalized adjustment input. Users can provide feedback on the effect of the action generation, such as "The leg lift is too difficult, please reduce it a bit" or "The hold time is too short, I hope it will be more challenging." The system receives this feedback through the input / output interface and stores it in the database for subsequent processing. Specifically, the user adjusts the parameters of the action based on the feedback, such as the leg lift angle or hold time, which is represented as a vector ΔP.

[0153] (2) Adjust the action generation strategy. The processor adjusts the text feature and body feature encoding based on user feedback. The program can increase the holding time of the action, reduce the angle of the leg lift, or adjust the intensity of balance control. The adjusted features will be re-input into the diffusion model to generate an improved action sequence, thereby providing users with fitness guidance that better meets their needs.

[0154] Adjust the action generation strategy: Adjust the text features and body features based on the feedback:

[0155] F T ′=F T +ΔP;

[0156] Among them, F T ′ is the action after adjustment; F T To adjust the previous action.

[0157] Re-input the diffusion model to generate new motion latent features and decode to generate action sequences, repeating steps 3 and 4.

[0158] Step 9: Movement Improvement and Repetition.

[0159] (1) Repeated generation and action optimization.

[0160] Based on user feedback, the system can repeat the process of action generation and ultimately provide the most suitable fitness actions for users by continuously adjusting the details of the generation process.

[0161] This iterative process of action generation and user feedback makes the training process more personalized, and each user can get training content that suits their physical condition and goals.

[0162] Repeated generation and action optimization: In each iteration, the action features are continuously updated and optimized based on the user's feedback and performance until the user is satisfied with the generated action sequence.

[0163] (2) Long-term motion data storage and analysis.

[0164] The system will also store every training data of the user in the database, including the features generated by each action, the user's feedback, the actual execution performance, etc.

[0165] This data can be used to analyze the user's progress and develop long-term exercise plans. For example, if the user gradually improves in balance training, the system will increase the difficulty of the training to maintain the challenge and training effect.

[0166] Long-term motion data storage and analysis: The action sequence M of each training user Feedback feedback The parameter adjustment ΔP is stored in the database for long-term motion analysis and optimization.

[0167] The virtual fitness coach system provided by the present invention has the following advantages

[0168] 1. Personalized customized training.

[0169] The system generates personalized fitness movements based on user input and real-time feedback. This capability allows each user to receive tailored guidance during their training process, meeting their personalized needs in terms of both exercise difficulty and movement details.

[0170] 2. Real-time feedback and action improvement.

[0171] Through cameras and sensors, the system can monitor the user's movements in real time and give specific adjustment suggestions. This not only helps users correct mistakes in time during training, but also makes the training process safer.

[0172] 3. Remote training support.

[0173] The system has a communication interface that can be connected to external terminals to support remote training. This is very convenient for users who do fitness training at home. Users can get guidance and feedback from virtual coaches through the Internet without geographical restrictions.

[0174] 4. Improve the quality of actions through multiple iterations.

[0175] Through continuous movement generation and user feedback, the system can gradually optimize the generated fitness movements. The user's training data will be used for future movement improvements to provide more accurate and effective guidance.

[0176] Example 3

[0177] An electronic device, comprising:

[0178] One or more processors.

[0179] A memory for storing one or more programs.

[0180] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of generating human body motion from fine-grained text in Example 1.

[0181] Example 4

[0182] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of generating human body motion from fine-grained text in Example 1.

[0183] Example 5

[0184] A computer program product includes a computer program, which, when executed by a processor, implements the method of generating human body motion from fine-grained text in embodiment 1.

[0185] Example 6

[0186] A computer device, which may be a database, may have an internal structure as shown in Figure 2 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the method of generating human body motion from fine-grained text in Example 1 is implemented.

[0187] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0188] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0189] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for generating human motion from fine-grained text, characterized in that: The method comprises: Obtaining relevant information of the human body movement to be generated; the relevant information includes operation information, fine-grained text condition information and spatial control signal information; Encoding the relevant information to obtain features of the relevant information; De-noising the features of the relevant information to obtain motion potential features; Decoding the motion potential features to obtain a running sequence; The operation sequence is post-processed to obtain operation information of the human body movement to be generated.

2. The method for generating human body motion from fine-grained text according to claim 1, characterized in that: A pre-trained CLIP text encoder is applied to encode the fine-grained textual conditional information.

3. The method for generating human body motion from fine-grained text according to claim 1, characterized in that: A pre-trained multi-layer neural network is used to encode the spatial control signal information.

4. The method for generating human body motion from fine-grained text according to claim 1, characterized in that: A diffusion model is applied to denoise the features of the relevant information; the denoising process of the diffusion model includes a forward denoising process and a reverse denoising process.

5. The method for generating human body motion from fine-grained text according to claim 4, characterized in that: The forward noise addition process follows a Markov chain.

6. The method for generating human body motion from fine-grained text according to claim 5, characterized in that: The Markov chain is: Among them, z n is the potential noise feature in the process of adding noise in the nth step; z n-1 is the potential noise feature in the process of adding noise in the n-1th step; α n It is a hyperparameter used to adjust the degree of noise addition; I is the unit matrix; is the normal distribution; q is the conditional probability distribution.

7. A device for generating human body motion from fine-grained text, characterized in that: The device comprises: An input module, used to obtain relevant information of the human body movement to be generated; the relevant information includes operation information, fine-grained text condition information and space control signal information; An encoding module, used to encode the relevant information to obtain features of the relevant information; A denoising module, used for denoising the features of the relevant information to obtain motion potential features; A decoding module, used for decoding the motion potential features to obtain a running sequence; The post-processing module is used to post-process the operation sequence to obtain the operation information of the human body movement to be generated.

8. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating human body motion from fine-grained text as described in any one of claims 1-6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating human body motion from fine-grained text as described in any one of claims 1-6 is implemented.

Images