Method, system and device for generating three-dimensional motion based on voice input

Abstract

The present invention belongs to the field of artificial intelligence and machine learning, and specifically relates to a method, system and device for generating three-dimensional actions based on voice input, aiming to solve the problem that the existing method of converting voice commands into 3D actions is not effective. The present invention includes: converting audio data into text form, and performing text parsing and command verification to obtain text data; tokenizing and word embedding the text data, adding position encoding to form a dimensional position vector and inputting it into the Transformer model to generate an action description sequence; evaluating the coherence of the action sequence in the context through a coherence score function and setting a coherence threshold to filter the action sequence to obtain a high-quality action series; based on the model, performing noise reduction, posture capture and time capture on the high-quality action series to finally obtain a 3D motion sequence. The present invention can ensure high-accuracy parsing and action generation for various complex instructions.

Description

Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, natural language processing (NLP) and computer vision (CV) have been widely used in many fields, especially in achieving human-computer interaction. In particular, voice-controlled 3D motion generation technology has shown great potential and value in the fields of virtual reality (VR), augmented reality (AR), games, robot navigation, and assisting people with disabilities. 3D motion generation technology can be applied to virtual assistants, interactive games, virtual reality training simulators, smart home control, and applications that provide assistance to people with visual or motor impairments. In these scenarios, users can interact with the system through simple voice commands, and the system can understand these commands and generate corresponding actions or responses, greatly improving the user experience and operational convenience. At present, both speech recognition technology and motion generation technology have made significant progress. In particular, the application of deep learning has greatly improved the accuracy of speech recognition, and the motion generation model based on deep learning can also generate more and more realistic movements. However, there are still certain challenges in seamlessly converting voice commands into 3D action sequences, especially in complex environments and changing contexts. Although existing technologies can handle simple commands, it is still a challenge to understand complex voice commands that contain multiple actions or need to be executed in a specific context. Although existing action generation models can generate actions, it is still somewhat difficult to achieve highly realistic and accurate actions that match specific voice commands, especially in terms of action details and coherence. Existing systems often ignore contextual information when understanding instructions, resulting in the generated actions being less than ideal in terms of logic or scene adaptability. It is still a relatively difficult task for existing technologies to effectively process and utilize large-scale data sets for model training, especially when they contain complex action descriptions and video data.

[0003] Therefore, the existing method of converting voice commands into 3D actions is not effective. Summary of the invention

[0004] In order to solve the above-mentioned problem in the prior art, that is, the problem that the prior art method of converting voice commands into 3D actions is not effective, the present invention provides a method for generating three-dimensional actions based on voice input, the method comprising:

[0005] Acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data;

[0006] Tokenizing the text data, dividing the text data into token units, converting each token unit into a fixed-dimensional vector through word embedding, adding position encoding to each fixed-dimensional vector to form a dimensional position vector; inputting all the position dimensional vectors into a Transformer model to generate an action description sequence;

[0007] Based on the action description sequence, the probability of different actions appearing in the context is calculated through a conditional probability model; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to filter the action sequence, and after filtering, a high-quality action series is obtained by combining;

[0008] The diffusion model based on the transformer framework performs noise reduction, posture capture and time capture on the high-quality action series to finally obtain a 3D motion sequence

[0009] The method of adding position encoding to each fixed-dimensional vector to form a dimension position vector is:

[0010] X i =E(x i )+P i ;

[0011] Among them, P i is the position code calculated based on position i, E(x i ) is a vector of fixed dimension, X i is the dimension position vector, x i for.

[0012] The method to calculate the probability of different actions appearing in the context through the conditional probability model is:

[0013]

[0014] Among them, P(a|c) represents the probability of action a given condition variable c, W a and b a are the weight vector and bias term associated with action a, c represents the conditional vector for a given condition, and c T represents the transpose of c, W d and b d They represent the weight vector and bias term associated with action b respectively, A represents the set of all actions, and a represents an action.

[0015] The method to evaluate the coherence of action sequences in context is:

[0016]

[0017] Among them, S(a,c) represents the coherence score of action a in the entire action sequence in a given condition variable c, n is the number of actions in the sequence, i represents the index, and P(a i |c i ) indicates that the given condition c i In the case of action a i The conditional probability of occurrence.

[0018] The process of generating an action description sequence from text data also includes using an update function to update the context at any time. The method of updating the context at any time is specifically as follows:

[0019] c new =f(c old ,a prev ,env);

[0020] Among them, f is the update function, a prev is the previous action, env represents the current environment information, c old Represents the context before update, c new Represents the updated context.

[0021] The method for performing noise reduction processing on the high-quality action series includes:

[0022] Introducing Gaussian noise into the latent code z in the high-quality action series generates perturbed samples:

[0023]

[0024] Among them, ε is a constant that is not equal to 0, As a noise scheduler, t represents the diffusion time step, z t represents the disturbance sample at time t;

[0025] The disturbance samples are subjected to noise reduction processing:

[0026]

[0027] Among them, p θ represents a parameterized probability distribution function, where the parameter set is θ, z t represents the disturbance sample at time t, z t-1 represents the denoised sample at time t-1, μ θ (z t ) indicates that the mean of the normal distribution is z t The function of is represented by the parameter θ;

[0028] Right t-1 After t-1 times of denoising, the denoising model z0 is obtained.

[0029] The 3D motion sequence is obtained by:

[0030] The posture information is obtained by performing posture capture on the high-quality action series through the posture transformer block:

[0031] P i =T pose (P i-1 ,C i );

[0032] Among them, P i is the pose encoding of time step i, C i P is the context information containing the speech instruction encoding. i-1 represents the posture at the i-1th time point, T pose A representation is a transformation that takes as input the pose at the previous time point and the condition at the current time point, and outputs the pose at the current time point.

[0033] The high-quality action series is temporally captured through the temporal transformer block to obtain temporal encoding:

[0034] T i =T time (T i-1 ,Δt);

[0035] Among them, T i is the time code of time step i, Δt represents the time interval, T time Represents a function that uses the previous state T i-1 and time state increment Δt to generate the current state T i , T i-1 Represented as the time encoding of time step i-1.

[0036] Output 3D motion sequence Z based on posture information, temporal encoding and denoising model of high-quality action series i :Z i =D(z0,T i ,P i ).

[0037] The method for generating three-dimensional motion based on speech input also includes a process of training the Transformer model based on training data, specifically including:

[0038] The model parameters are trained by maximizing the log-likelihood of observed action command and context pairs:

[0039]

[0040] Where D is the training dataset containing all observed action and context pairs, W*b* represents the value of the optimal parameter, and argmax W,b It means to find the values ​​of W and b that can maximize the following expression. a represents the action, c represents the conditional vector of the given condition, and P represents the probability.

[0041] In the process of denoising the high-quality action sequences of the training data, the log-likelihood of the variational lower bound is reduced to:

[0042] L θ = -log p θ (z t-1 |z t )+

[0043] ∑ t D KL (q(z t-1 |z t ,z0)||p θ (z t-1 |z t ));

[0044] Among them, L θ It is the loss function about the parameter θ, which usually needs to be minimized. KL It is the Kullback-Leibler divergence, which measures the difference between two probability distributions, p θ It represents the conditional probability distribution defined by the model, q(z t-1 |z t , z0) represents a variational distribution used to approximate the true distribution p θ (z t-1 |z t ), z t represents the disturbance sample at time t; z t-1 represents the perturbation sample at time t-1, and z0 represents the denoising model.

[0045] Optimize the Transformer model by minimizing the log-likelihood of the variational lower bound;

[0046] In the denoising process of the training data, the Transformer model is trained by a loss function to obtain denoised data close to the original data; the loss function is:

[0047]

[0048] Among them, L simple represents a simplified version of the loss function, which is used in the training phase to evaluate and optimize the model performance. x~p(x) represents the sample x drawn from the probability distribution p(x). Represents the model parameterization, defined by the parameter θ, the error prediction function, x represents the sample, p(x) represents the probability distribution, and ε is a non-zero constant.

[0049] A second aspect of the present invention provides a system for generating three-dimensional motions based on voice input, characterized in that the system comprises:

[0050] A data acquisition module is used to acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data;

[0051] A sequence generation module is used to tokenize the text data, divide the text data into token units, convert each token unit into a fixed-dimensional vector through word embedding, add position encoding to each fixed-dimensional vector to form a dimensional position vector; input all position dimensional vectors into a Transformer model to generate an action description sequence;

[0052] An action prediction and screening module is used to calculate the probability of different actions appearing in the context through a conditional probability model based on the action description sequence; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to screen the action sequence, and after screening, a high-quality action series is obtained by combining;

[0053] The motion sequence generation module is used to perform noise reduction, posture capture and time capture on the high-quality action series based on the diffusion model of the transformer framework to finally obtain a 3D motion sequence.

[0054] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0055] at least one processor; and

[0056] a memory communicatively connected to at least one of the processors; wherein,

[0057] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method of generating three-dimensional actions based on voice input.

[0058] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned method of generating three-dimensional actions based on voice input.

[0059] Beneficial effects of the present invention:

[0060] (1) Using the Transformer model to extract action descriptions This paper uses the Transformer model to extract action descriptions from text, which can effectively understand and extract the action details and sequence in complex instructions. This method improves the conversion efficiency and accuracy from text to action descriptions, laying a solid foundation for subsequent action generation;

[0061] (2) Through context analysis, the system can improve its ability to understand action instructions in different scenarios. This not only enhances the model's adaptability to context changes, but also ensures the logical and contextual coherence of the generated action sequence, significantly improving the accuracy of action instruction parsing and the naturalness of action generation;

[0062] (3) A diffusion model is used to generate 3D action sequences based on the determined action descriptions. The diffusion model, with its powerful data generation capabilities, is able to generate highly realistic actions, which not only accurately conforms to the user's instructions, but also makes the generated actions more natural and smooth visually. Emphasis is placed on building a dataset containing detailed action descriptions and high-quality videos, and using these data for model training. By ensuring the diversity, balance, and consistency of the dataset, this method can effectively improve the generalization ability and performance of the model, ensuring high-accuracy parsing and action generation for various complex instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0064] Figure 1 is a flow chart of a method for generating three-dimensional actions based on voice input according to an embodiment of the present invention;

[0065] Figure 2 It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION

[0066] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0067] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0068] The present invention provides a method for generating three-dimensional motion based on voice input, the method comprising:

[0069] Acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data;

[0070] Tokenizing the text data, dividing the text data into token units, converting each token unit into a fixed-dimensional vector through word embedding, adding position encoding to each fixed-dimensional vector to form a dimensional position vector; inputting all the position dimensional vectors into a Transformer model to generate an action description sequence;

[0071] Based on the action description sequence, the probability of different actions appearing in the context is calculated through a conditional probability model; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to filter the action sequence, and after filtering, a high-quality action series is obtained by combining;

[0072] The diffusion model based on the transformer framework performs noise reduction, posture capture and time capture on the high-quality action series to finally obtain a 3D motion sequence.

[0073] In order to more clearly explain the method for generating three-dimensional actions based on voice input of the present invention, Figure 1 Each step in the embodiment of the present invention is described in detail.

[0074] The method for generating three-dimensional motion based on voice input in the first embodiment of the present invention, each step is described in detail as follows:

[0075] Acquire audio data including voice instructions, convert the audio data into text form using sound recognition technology, and perform text parsing and instruction verification to obtain text data; in this embodiment, specifically, receive audio data of voice instructions provided by the user, and convert it into text format using speech recognition technology. This process first includes the operation of converting voice signals into digital signals, and then converting these digital signals into understandable text representations by using a deeply trained speech recognition model. This conversion is not only based on the physical characteristics of the sound, but also involves the understanding of language patterns, contexts, and possible accents. Parse the converted text to identify the commands and parameters contained therein. Analyze the grammatical structure and semantic content in the text to accurately understand the user's intentions. Verify the validity of the converted instructions, and provide feedback to the user when necessary to correct possible errors or unclear instructions. This step ensures that the system can correctly understand and execute the user's instructions, thereby improving the efficiency and accuracy of the entire 3D action generation.

[0076] In this embodiment, the audio data should contain action instructions described in natural language, and the text data should be detailed enough to guide the specific details of the action. Specifically include descriptions such as direction, action type, speed, duration, etc. Example: "Walk to the door, stop, then turn right and run to the window." For video data, the video should accurately show the action sequence in the text description. Every detail of the action needs to be captured to ensure that it matches the text description. The video quality should be high enough to capture the subtle changes and key points of the action.

[0077] In this embodiment, the audio data should be diverse and cover multiple action types and various complex instruction combinations to ensure that the model can handle a variety of possible inputs. The dataset should balance different types of actions and descriptions to avoid over-representation of a certain category. The consistency between the text description and the video is crucial, and any text description should have an accurate corresponding video display. The actions in the video should be performed by real humans to ensure that the generated actions are realistic and credible. Manual or semi-automated tools are used to annotate the video data to match each action with the instructions in the text description. The annotations can include timestamps for the start and end of the action, the type of action, the subject performing the action, etc.

[0078] The text data is tokenized, the text data is divided into tag units, each tag unit is converted into a fixed-dimensional vector through word embedding, and a position code is added to each fixed-dimensional vector to form a dimensional position vector; all position dimensional vectors are input into the Transformer model to generate an action description sequence; in this embodiment, specifically, the text data first needs to be tokenized (Tokenization), which divides the text into smaller units (e.g., words, characters, or subwords). For example, the sentence "Take two steps forward" can be divided into `["forward", "walk", "two steps"]`. Then, each tag is converted into a fixed-dimensional vector through a word embedding process and then input into the Transformer model. The position code is then added to each word embedding vector to provide the model with information about the position of the word in the sequence:

[0079] In an embodiment of the present invention, the method of adding position coding to each fixed-dimensional vector to form a dimensional position vector is:

[0080] X i =E(x i )+P i ;

[0081] Among them, P i is the position code calculated based on position i, E(x i ) is a vector of fixed dimension, X i is the dimension position vector, x iis the original input representing the i-th element.

[0082] Based on the action description sequence, the probability of different actions appearing in the context is calculated through a conditional probability model; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to filter the action sequence, and after filtering, a high-quality action series is obtained by combining;

[0083] In an embodiment of the present invention, the method for calculating the probability of different actions appearing in a context by a conditional probability model is:

[0084]

[0085] Among them, P(a|c) represents the probability of action a given condition variable c, W a and b a are the weight vector and bias term associated with action a, c represents the conditional vector for a given condition, and c T represents the transpose of c, W d and b d They represent the weight vector and bias term associated with action b respectively, A represents the set of all actions, and a represents an action.

[0086] Through this probabilistic model, the model can capture the given context information and predict the next action based on it. Using a probabilistic model can help the model understand and learn the text in the sequence data, which helps the model to better predict and generalize when faced with new and unseen data. This formula trains the model parameters by maximizing the log-likelihood of the observed action instruction and context pairs.

[0087] In an embodiment of the present invention, the method for evaluating the coherence of an action sequence in a context by using a coherence score function is:

[0088]

[0089] Among them, S(a,c) represents the coherence score of action a in the entire action sequence in a given condition variable c, n is the number of actions in the sequence, i represents the index, and P(a i |c i ) indicates that the given condition c i In the case of action a i The conditional probability of occurrence.

[0090] In this embodiment, through the above formula, the context analysis module can parse complex action instructions more accurately and provide support for the generation of subsequent actions. Such context analysis can significantly improve the performance of the action generation system, making the generated actions more realistic and logical. During the training process, the model will try to optimize the parameters to produce outputs with higher coherence scores, which can reduce the generation of incoherent or semantically unreasonable texts. High scores usually mean that the output is more compatible with the given context, so the output quality is also higher. We can set a threshold so that when the model generates or selects an answer, it will only accept the output when the coherence score is higher than this threshold. This helps to ensure that the output of the model meets certain quality standards.

[0091] In this embodiment, complex action descriptions, such as "walk to the door and sit down", are deeply parsed. Through transform processing technology, especially methods that rely on syntactic analysis and semantic understanding, we identify and annotate the verbs and related objects contained therein, thereby refining complex action instructions into orderly action components. For example, "walk to the door and sit down" is parsed into two actions, "walk to the door" and "sit down". After action classification and division, each simple action will be regarded as an input unit of the action sequence generation process. These units are combined to form a complete and executable action sequence. At this time, consider the logical and temporal relationship between actions, optimize the coherence and logic of the action sequence, and lay a solid foundation for generating accurate and natural action sequences. When dividing actions, the contextual relationship and coherence between actions must also be considered. For example, the action "walk to the door" followed by "sit down" should have a reasonable connection in physical space and action logic. Through context analysis, it is ensured that the generated action sequence is not only coherent in action, but also reasonable in story narration or scene logic.

[0092] Use the update function to update the context at any time. The specific method for updating the context at any time is:

[0093] c new =f(c old ,a prev ,env);

[0094] Among them, f is the update function, a prev is the previous action, env represents the current environment information, c old Represents the context before update, c new Represents the updated context.

[0095] The diffusion model based on the transformer framework performs noise reduction, posture capture and time capture on the high-quality action series to finally obtain a 3D motion sequence.

[0096] In this embodiment, a model framework is established based on the determined action category, and a diffusion model based on a transformer framework is used to generate the corresponding 3D motion sequence. The skeleton of this new type of diffusion model consists of two different transformer blocks, one is the posture transformer block (T pose ) The other is the temporal transformer block (T time ), the former focuses on the supervision of motion posture, and the latter is to capture time information.

[0097] The method for performing noise reduction processing on the high-quality action series includes:

[0098] Introducing Gaussian noise into the latent code z in the high-quality action series generates perturbed samples:

[0099]

[0100] Among them, ε is a constant that is not 0, As a noise scheduler, t represents the diffusion time step, z t represents the disturbance sample at time t;

[0101] The disturbance samples are subjected to noise reduction processing:

[0102]

[0103] Among them, p θ represents a parameterized probability distribution function, where the parameter set is θ, z t represents the disturbance sample at time t, z t-1 represents the denoised sample at time t-1, μ θ (z t ) indicates that the mean of the normal distribution is z t The function of is represented by the parameter θ;

[0104] Right t-1 After t-1 times of denoising, the denoising model z0 is obtained.

[0105] The 3D motion sequence is obtained by:

[0106] The posture information is obtained by performing posture capture on the high-quality action series through the posture transformer block:

[0107] P i =T pose (P i-1 ,C i );

[0108] Among them, P i is the pose encoding of time step i, Ci P is the context information containing the speech instruction encoding. i-1 represents the posture at the i-1th time point, T pose A representation is a transformation that takes as input the pose at the previous time point and the condition at the current time point, and outputs the pose at the current time point.

[0109] The high-quality action series is temporally captured through the temporal transformer block to obtain temporal encoding:

[0110] T i =T time (T i-1 ,Δt);

[0111] Among them, T i is the time code of time step i, Δt represents the time interval, T time Represents a function that uses the previous state T i-1 and time state increment Δt to generate the current state T i , T i-1 Represented as the time encoding of time step i-1.

[0112] Output 3D motion sequence Z based on posture information, temporal encoding and denoising model of high-quality action series i :Z i =D(z0,T i ,P i ).

[0113] The method for generating three-dimensional motion based on speech input also includes a process of training the Transformer model based on training data, specifically including:

[0114] The Transformer model is trained with a large amount of labeled text data to understand the structure and semantics of the language. The training goal of the model is to minimize the cross entropy loss between the predicted and actual labels:

[0115] Among them, y is the true label, is the predicted output of the model.

[0116] Cross entropy loss is a loss function commonly used in classification tasks. It measures the difference between two probability distributions, namely the distribution predicted by the model and the distribution of the actual labels.

[0117] The method for generating three-dimensional motion based on speech input also includes a process of training the Transformer model based on training data, specifically including:

[0118] The model parameters are trained by maximizing the log-likelihood of observed action command and context pairs:

[0119]

[0120] Where D is the training dataset containing all observed action and context pairs, W * b * Represents the value of the optimal parameter, argmax W,b It means to find the values ​​of W and b that can maximize the following expression. a represents the action, c represents the conditional vector of the given condition, and P represents the probability.

[0121] In the process of denoising the high-quality action sequences of the training data, the log-likelihood of the variational lower bound is reduced to:

[0122] L θ = -log p θ (z t-1 |z t )+∑ t D KL (q(z t-1 |z t ,z0)||p θ (z t-1 |z t ));

[0123] Among them, L θ It is the loss function about the parameter θ, which usually needs to be minimized. KL It is the Kullback-Leibler divergence, which measures the difference between two probability distributions, p θ It represents the conditional probability distribution defined by the model, q(z t-1 |z t , z0) represents a variational distribution used to approximate the true distribution p θ (z t-1 |z t ), z t represents the disturbance sample at time t; z t-1 represents the perturbation sample at time t-1, and z0 represents the denoising model.

[0124] Optimize the Transformer model by minimizing the log-likelihood of the variational lower bound;

[0125] In the denoising process of the training data, the Transformer model is trained by a loss function to obtain denoised data close to the original data; the loss function is:

[0126]

[0127] Among them, L simple represents a simplified version of the loss function, which is used in the training phase to evaluate and optimize the model performance. x~p(x) represents the sample x drawn from the probability distribution p(x). Represents the model parameterization, defined by the parameter θ, the error prediction function, x represents the sample, p(x) represents the probability distribution, and ε is a non-zero constant.

[0128] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0129] A system for generating three-dimensional motions based on voice input according to a second embodiment of the present invention comprises:

[0130] A data acquisition module is used to acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data;

[0131] A sequence generation module is used to tokenize the text data, divide the text data into token units, convert each token unit into a fixed-dimensional vector through word embedding, add position encoding to each fixed-dimensional vector to form a dimensional position vector; input all position dimensional vectors into a Transformer model to generate an action description sequence;

[0132] An action prediction and screening module is used to calculate the probability of different actions appearing in the context through a conditional probability model based on the action description sequence; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to screen the action sequence, and after screening, a high-quality action series is obtained by combining;

[0133] The motion sequence generation module is used to perform noise reduction, posture capture and time capture on the high-quality action series based on the diffusion model of the transformer framework to finally obtain a 3D motion sequence.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0135] It should be noted that the system for generating three-dimensional actions based on voice input provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations of the present invention.

[0136] An electronic device according to a third embodiment of the present invention includes:

[0137] at least one processor; and

[0138] a memory communicatively connected to at least one of the processors; wherein,

[0139] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method of generating three-dimensional actions based on voice input.

[0140] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned method of generating three-dimensional actions based on voice input.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the storage device and processing device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0142] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0143] Reference below Figure 2 , which shows a schematic diagram of the structure of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 2 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 2 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0145] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0146] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0150] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0151] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for generating three-dimensional motion based on voice input, characterized in that: The method comprises: Acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data; Tokenizing the text data, dividing the text data into token units, converting each token unit into a fixed-dimensional vector through word embedding, adding position encoding to each fixed-dimensional vector to form a dimensional position vector; inputting all the position dimensional vectors into a Transformer model to generate an action description sequence; Based on the action description sequence, the probability of different actions appearing in the context is calculated through a conditional probability model; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to filter the action sequence, and after filtering, a high-quality action series is obtained by combining; The diffusion model based on the transformer framework performs noise reduction, posture capture and time capture on the high-quality action series to finally obtain a 3D motion sequence; The method for performing noise reduction processing on the high-quality action series includes: Introducing Gaussian noise into the latent code z in the high-quality action series generates perturbed samples: Among them, ε is a constant that is not 0, As a noise scheduler, t represents the diffusion time step, z t represents the disturbance sample at time t; The disturbance samples are subjected to noise reduction processing: Among them, p θ represents a parameterized probability distribution function, where the parameter set is θ, z t represents the disturbance sample at time t, z t-1 represents the denoised sample at time t-1, μ θ (z t ) indicates that the mean of the normal distribution is z t The function of is represented by the parameter θ; Right t-1 After t-1 times of denoising, the denoising model z0 is obtained; The posture information is obtained by performing posture capture on the high-quality action series through the posture transformer block: P i =T pose (P i-1 ,C i ); Among them, P i is the pose encoding at time step i, C i P is the context information containing the speech instruction encoding. i-1 represents the posture at the i-1th time point, T pose The representation is a conversion; The high-quality action series is temporally captured through the temporal transformer block to obtain temporal encoding: T i =T time (T i-1 ,Δt); Among them, T i is the time code of time step i, Δt represents the time interval, T time Represents a function that uses the previous state T i-1 and time state increment Δt to generate the current state T i , T i-1 Represented as the time encoding of time step i-1; Output 3D motion sequence Z based on posture information, temporal encoding and denoising model of high-quality action series i :Z i =D(z0,T i ,P i ).

2. The method for generating three-dimensional motion based on voice input according to claim 1, characterized in that: The method of adding position encoding to each fixed-dimensional vector to form a dimension position vector is: X i =E(x i )+P i ; Among them, P i is the position code calculated based on position i, E(x i ) is a vector of fixed dimension, X i is the dimension position vector, x i is the original input representing the i-th element.

3. The method for generating three-dimensional motion based on voice input according to claim 2, characterized in that: The method to calculate the probability of different actions appearing in the context through the conditional probability model is: Among them, P(a|c) represents the probability of action a given condition variable c, W a and b a are the weight vector and bias term associated with action a, c represents the conditional vector for a given condition, and c T represents the transpose of c, W d and b d They represent the weight vector and bias term associated with action b respectively, A represents the set of all actions, and a represents an action.

4. The method for generating three-dimensional motion based on voice input according to claim 3, characterized in that: The method to evaluate the coherence of action sequences in context is: Among them, S(a,c) represents the coherence score of action a in the entire action sequence in a given condition variable c, n is the number of actions in the sequence, i represents the index, and P(a i |c i ) indicates that the given condition c i In the case of action a i The conditional probability of occurrence.

5. The method for generating three-dimensional motion based on voice input according to claim 4, characterized in that: The process of generating an action description sequence from text data also includes using an update function to update the context at any time. The method of updating the context at any time is specifically as follows: c new =f(c old ,a prev,env ); Among them, f is the update function, a prev is the previous action, env represents the current environment information, c old Represents the context before update, c new Represents the updated context.

6. The method for generating three-dimensional motion based on voice input according to claim 5, characterized in that: The method for generating three-dimensional motion based on speech input also includes a process of training the Transformer model based on training data, specifically including: The model parameters are trained by maximizing the log-likelihood of observed action command and context pairs: Where D is the training dataset containing all observed action and context pairs, W * b * Represents the value of the optimal parameter, argmax W,b It means to find the value of W and b that can maximize the following expression. a represents the action, c represents the conditional vector of the given condition; P represents In the process of denoising the high-quality action sequences of the training data, the log-likelihood of the variational lower bound is reduced to: Among them, L θ It is the loss function about the parameter θ, which usually needs to be minimized. K It is the Kullback-Leibler divergence, which measures the difference between two probability distributions, p θ It represents the conditional probability distribution defined by the model, q(z t-1 |z t , z0) represents a variational distribution used to approximate the true distribution p θ (z t-1 |z t ), z t represents the disturbance sample at time t; z t-1 represents the disturbance sample at time t-1, and z0 represents the denoising model; Optimize the Transformer model by minimizing the log-likelihood of the variational lower bound; In the denoising process of the training data, the Transformer model is trained by a loss function to obtain denoised data close to the original data; the loss function is: Among them, L simple represents a simplified version of the loss function, which is used in the training phase to evaluate and optimize the model performance. x~p(x) represents the sample x drawn from the probability distribution p(x). Represents the model parameterization, defined by the parameter θ, the error prediction function, x represents the sample, p(x) represents the probability distribution, and ε is a non-zero constant.

7. A system for generating three-dimensional motion based on voice input, characterized in that: The system comprises: A data acquisition module is used to acquire audio data including voice commands, convert the audio data into text form using voice recognition technology, and perform text parsing and command verification to obtain text data; A sequence generation module is used to tokenize the text data, divide the text data into token units, convert each token unit into a fixed-dimensional vector through word embedding, add position encoding to each fixed-dimensional vector to form a dimensional position vector; input all position dimensional vectors into a Transformer model to generate an action description sequence; An action prediction and screening module is used to calculate the probability of different actions appearing in the context through a conditional probability model based on the action description sequence; according to the probability of different actions appearing in the context, the coherence of the action sequence in the context is evaluated through a coherence score function and a coherence threshold is set to screen the action sequence, and after screening, a high-quality action series is obtained by combining; A motion sequence generation module is used to perform noise reduction, posture capture and time capture on the high-quality action series based on a diffusion model of a transformer framework to finally obtain a 3D motion sequence; The method for performing noise reduction processing on the high-quality action series includes: Introducing Gaussian noise into the latent code z in the high-quality action series generates perturbed samples: Among them, ε is a constant that is not 0, As a noise scheduler, t represents the diffusion time step, z t represents the disturbance sample at time t; The disturbance samples are subjected to noise reduction processing: Among them, p θ represents a parameterized probability distribution function, where the parameter set is θ, z t represents the disturbance sample at time t, z t-1 represents the denoised sample at time t-1, μ θ (z t ) indicates that the mean of the normal distribution is z t The function of is represented by the parameter θ; Right t-1 After t-1 times of denoising, the denoising model z0 is obtained; The posture information is obtained by performing posture capture on the high-quality action series through the posture transformer block: P i =T pose (P i-1 ,C i ); Among them, P i is the pose encoding of time step i, C i P is the context information containing the speech instruction encoding. i-1 represents the posture at the i-1th time point, T pose The representation is a conversion; The high-quality action series is temporally captured through the temporal transformer block to obtain temporal encoding: T i =T time (T i-1 ,Δt); Among them, T i is the time code of time step i, Δt represents the time interval, T time Represents a function that uses the previous state T i-1 and time state increment Δt to generate the current state T i , T i-1 Represented as the time encoding of time step i-1; Output 3D motion sequence Z based on posture information, temporal encoding and denoising model of high-quality action series i :Z i =D(z0,T i ,P i ).

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the method for generating three-dimensional actions based on voice input as described in any one of claims 1-7.

Images