Humanoid robot posture generation method and device, robot, medium and product

Abstract

The present application relates to the technical field of robots, and discloses a posture generation method and device for a humanoid robot, the robot, a medium and a product, the method comprising: obtaining first voice data to be output, a first special action mark, Gaussian noise and a time step; determining a first mel-frequency spectrum and a first deep semantic feature according to the first voice data, the first deep semantic feature being the first voice data processed by a self-supervised voice representation learning model; inputting the first mel-frequency spectrum, the first deep semantic feature, the first special action mark, the Gaussian noise and the time step into a posture generation model, determining first action data according to the output of the posture generation model, the posture generation model being a diffusion model that has completed training; and synchronously controlling the humanoid robot to generate a gesture action represented by the first action data when the first voice data is output. The present application can generate more diverse actions within a distribution, and improve the naturalness and closeness of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method, apparatus, robot, medium, and product for generating poses of a humanoid robot. Background Technology

[0002] Humanoid robots are robots that mimic human appearance and behavior and possess a high level of intelligence. Compared with traditional industrial robots and service robots, humanoid robots have similar perception methods, "limb" structures, and movement patterns to humans, playing an important role in applications such as manufacturing, social services, and special operations.

[0003] Humanoid robots typically possess the ability to interact with humans in real time, quickly understanding and effectively executing commands given by humans through language, gestures, and other means. To enhance the interactive experience, humanoid robots often employ hand gestures when interacting with humans. However, these gestures are generally delivered through pre-recorded playback or remote control, resulting in fixed, unautonomous movements and a low degree of naturalness in human-robot interaction. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, robot, medium and product for generating poses of a humanoid robot, in order to solve the problem that existing humanoid robots lack autonomy and have low naturalness in human-computer interaction.

[0005] In a first aspect, the present invention provides a pose generation method for a humanoid robot, the method comprising: acquiring first speech data to be output, a first special action marker, Gaussian noise, and a time step, wherein the first special action marker is used to characterize the style of the generated gesture or a preset special action; determining a first Mel spectrum and a first deep semantic feature based on the first speech data, wherein the first deep semantic feature is the first speech data processed by a self-supervised speech representation learning model; inputting the first Mel spectrum, the first deep semantic feature, the first special action marker, Gaussian noise, and the time step into a pose generation model, and determining first action data based on the output of the pose generation model, wherein the pose generation model is a trained diffusion model; and synchronously controlling the humanoid robot to generate the gesture represented by the first action data while outputting the first speech data.

[0006] The pose generation model in this embodiment is a pre-trained diffusion model. Due to the randomness of the initial noise in the model, the randomness and diversity of generated actions can be improved, making the humanoid robot more human-like and enhancing the naturalness and intimacy of human-computer interaction. Furthermore, adding a first special action marker to the input enables the output of special actions even when the first speech data is empty audio, ensuring that gesture generation maintains reasonable performance even without speech output. In addition, converting the first speech data into first Mel spectrum and first deep semantic features before input not only enables the pose generation model to generate reasonable actions more accurately and improves generalization ability, but also reduces the model's computational load and improves action generation efficiency.

[0007] In an optional implementation, before inputting the first Mel spectrum, first deep semantic features, first special action marker, Gaussian noise, and time step into the pose generation model, the method further includes: constructing a pose generation model based on a sample database, wherein the sample database includes multiple training samples, the training samples include second speech data, second action data that follows the changes in the second speech data, and second special action marker, the duration of the second speech data being the same as the duration of the first speech data.

[0008] In this embodiment, before inputting the first Mel spectrum, the first deep semantic features, the first special action marker, Gaussian noise, and the time step into the pose generation model, a pose generation model is constructed using a sample database. The sample database contains diverse training samples, which allows the diffusion model to capture the mapping relationship between the pose and inputs such as the Mel spectrum, semantic features, and special action markers, thereby improving the accuracy of the generated pose.

[0009] In one optional implementation, constructing a pose generation model based on a sample database includes: extracting target training samples from the sample database, wherein the target training sample is one of multiple training samples, and the target training sample includes target speech data, target action data that changes with the target speech data, and target special action markers; determining the target Mel spectrum and target deep semantic features based on the target speech data; adding noise to the target action data according to a time step to obtain target noise data; and training a diffusion model based on the target Mel spectrum, target deep semantic features, target noise data, time step, target special action markers, and target action data to obtain the pose generation model.

[0010] In an optional implementation, before training the diffusion model, the method further includes: determining whether the target Mel spectrum includes 0; when the target Mel spectrum includes 0, mapping the region where the target Mel spectrum is 0 and the portion of the target deep semantic features corresponding to the region where the target Mel spectrum is 0 to learnable parameters to obtain updated target Mel spectrum and updated target deep semantic features; training the diffusion model based on the target Mel spectrum, target deep semantic features, target noise data, time step, target special action markers, and target action data, including: training the diffusion model based on the updated target Mel spectrum, updated target deep semantic features, target noise data, time step, target special action markers, and target action data.

[0011] In this embodiment, replacing the region with a Mel spectrum of 0 with a learnable parameter enables the diffusion model to fully associate empty audio with non-speaking actions. Thus, even if only a portion of a segment of length L is empty audio, the diffusion model can still generate non-speaking actions accordingly.

[0012] In one optional implementation, before training the diffusion model, the method further includes: determining whether the target Mel spectrum is all zero; when the target Mel spectrum is all zero, mapping the target Mel spectrum and the target deep semantic features to learnable parameters to obtain first model parameters and second model parameters; training the diffusion model based on the target Mel spectrum, target deep semantic features, target noise data, time step, target special action markers, and target action data, including: training the diffusion model based on the first model parameters, second model parameters, target noise data, time step, target special action markers, and target action data.

[0013] In this embodiment, replacing the all-zero Mel spectrum with learnable parameters enables the diffusion model to learn more coherent non-speaking state actions.

[0014] In an optional implementation, before extracting target training samples from the sample database, the method further includes: acquiring at least one third speech data and third action data corresponding to the third speech data, wherein the duration of the third speech data is greater than the duration of the second speech data; and slicing the at least one third speech data to obtain multiple training samples.

[0015] In one optional implementation, the third action data includes an action start phase, an action hold phase, and an action end phase. Before extracting target training samples from the sample database, the method further includes: setting the weight of the first training sample to a first preset value, setting the weight of the second training sample to a second preset value, and setting the weight of the third training sample to a third preset value. The first training sample is the training sample corresponding to the second action data including the action start phase, the second training sample is the training sample corresponding to the second action data including the action hold phase, and the third training sample is the training sample corresponding to the second action data including the action end phase. The first preset value is greater than the second preset value, and the second preset value is greater than the third preset value.

[0016] In this embodiment, by setting different weights for the training samples and increasing the weight of the training samples at the beginning of the movement, the pose generation model can be made to perform special actions with a greater probability in idle scenarios, while ensuring the continuity of the actions.

[0017] In one optional implementation, the first voice data is empty audio, and the first special action marker is determined according to the frequency of a preset special action.

[0018] Secondly, the present invention provides a posture generation device for a humanoid robot, the device comprising: an acquisition module for acquiring first speech data to be output, a first special action marker, Gaussian noise, and a time step, wherein the first special action marker is used to characterize the style of the generated gesture or a preset special action; a feature determination module for determining a first Mel spectrum and a first deep semantic feature based on the first speech data, wherein the first deep semantic feature is the first speech data processed by a self-supervised speech representation learning model; an action determination module for inputting the first Mel spectrum, the first deep semantic feature, the first special action marker, Gaussian noise, and the time step into a posture generation model, and determining first action data based on the output of the posture generation model; and an output module for synchronously controlling the humanoid robot to generate the gesture represented by the first action data while outputting the first speech data.

[0019] In one optional embodiment, the apparatus further includes: a construction module for constructing a pose generation model based on a sample database, wherein the sample database includes multiple training samples, the training samples include second speech data, second motion data that follows the changes in the second speech data, and a second special motion marker, the duration of the second speech data being the same as the duration of the first speech data.

[0020] In one optional implementation, the construction module includes: an extraction unit for extracting target training samples from a sample database, wherein the target training sample is one of a plurality of training samples, and the target training sample includes target speech data, target action data that changes with the target speech data, and target special action markers; a first determination unit for determining target Mel spectrum and target deep semantic features based on the target speech data; a noise addition unit for adding noise to the target action data according to a time step to obtain target noise data; and a training unit for training the diffusion model based on the target Mel spectrum, target deep semantic features, target noise data, time step, target special action markers, and target action data to obtain a pose generation model.

[0021] In an optional embodiment, the apparatus further includes: a first determining module, configured to determine whether the target Mel spectrum includes 0; a first mapping module, configured to, when the target Mel spectrum includes 0, map the region where the target Mel spectrum is 0 and the portion of the target deep semantic features corresponding to the region where the target Mel spectrum is 0 to learnable parameters, thereby obtaining an updated target Mel spectrum and updated target deep semantic features; the training unit includes: a first sub-training unit, configured to train the diffusion model based on the updated target Mel spectrum, the updated target deep semantic features, target noise data, time steps, target special action markers, and target action data.

[0022] In an optional embodiment, the apparatus further includes: a second determining module, configured to determine whether the target Mel spectrum is all zero; a second mapping module, configured to map the target Mel spectrum and target deep semantic features into learnable parameters when the target Mel spectrum is all zero, to obtain first model parameters and second model parameters; the training unit includes: a second sub-training unit, configured to train the diffusion model based on the first model parameters, the second model parameters, target noise data, time steps, target special action markers, and target action data.

[0023] In one optional embodiment, the apparatus further includes: a data acquisition module, configured to acquire at least one third voice data and third action data corresponding to the third voice data, wherein the duration of the third voice data is greater than the duration of the second voice data; and a slicing module, configured to slice the at least one third voice data to obtain multiple training samples.

[0024] In one optional implementation, the third action data includes an action start phase, an action hold phase, and an action end phase. The device further includes a processing module, configured to set the weight of the first training sample to a first preset value, set the weight of the second training sample to a second preset value, and set the weight of the third training sample to a third preset value. The first training sample is a training sample corresponding to the second action data including the action start phase, the second training sample is a training sample corresponding to the second action data including the action hold phase, and the third training sample is a training sample corresponding to the second action data including the action end phase. The first preset value is greater than the second preset value, and the second preset value is greater than the third preset value.

[0025] In one optional implementation, the first voice data is empty audio, and the first special action marker is determined according to the frequency of a preset special action.

[0026] Thirdly, the present invention provides a humanoid robot, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the posture generation method of the humanoid robot described in the first aspect or any corresponding embodiment.

[0027] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a humanoid robot to perform the posture generation method of the humanoid robot described in the first aspect or any corresponding embodiment thereof.

[0028] Fifthly, the present invention provides a computer program product, including computer instructions for causing a humanoid robot to execute the posture generation method of the humanoid robot described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a method for generating the pose of a humanoid robot according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a humanoid robot outputting hand gestures in an idle scenario according to an embodiment of the present invention; Figure 3This is a schematic diagram of a humanoid robot outputting hand gestures in an idle scenario according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating another method for generating the pose of a humanoid robot according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of constructing a pose generation model according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the diffusion model according to an embodiment of the present invention; Figure 7 This is a schematic diagram of third action data according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a partially substituted learnable parameter where the Mel spectrum is 0, according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a Mel spectrum processing procedure according to an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating the replacement of learnable parameters when the Mel spectrum is all 0 according to an embodiment of the present invention; Figure 11 This is a structural block diagram of a posture generation device for a humanoid robot according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the hardware structure of the humanoid robot according to an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] As mentioned in the background technology, the gestures output by humanoid robots when interacting with users are limited to pre-recorded actions or operator control, lacking autonomy, feeling mechanical, and the interaction experience appears stiff and not intimate enough.

[0033] In view of this, the present invention provides a method, apparatus, robot, medium and product for generating poses of a humanoid robot. A trained diffusion model (pose generation model) is configured on the humanoid robot, and motion data is obtained by using the pose generation model. Due to the randomness of the initial noise of the model, compared with recording, more diverse motions can be generated within the distribution, thereby improving the naturalness and intimacy of human-computer interaction.

[0034] The pose generation method for humanoid robots provided by the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a humanoid robot such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a pose generation method for humanoid robots, which can be used for humanoid robots. Figure 1 This is a flowchart illustrating a pose generation method for a humanoid robot according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step S101: Obtain the first speech data to be output, the first special action marker, Gaussian noise, and the time step.

[0036] The first special action marker is used to characterize the style of the generated gesture or a preset special action. The preset special action can be standing naturally with slight swaying, waving, liking, moving fingers, or looking down at hands, etc.

[0037] Specifically, in a dialogue scenario, the first special action marker is used to characterize the style of the gestures generated by the humanoid robot when outputting the first voice data. The style reflects the personality characteristics of the humanoid robot (such as lively or serious). The first special action marker can be pre-configured in the humanoid robot by the staff. In an idle (static) scenario, the first special action marker is used to characterize a certain special action generated by the humanoid robot when outputting the first voice data. Multiple special actions are pre-configured in the humanoid robot, and multiple special actions appear at a preset frequency. The first special action marker can be randomly generated according to the preset special action frequency.

[0038] For example, the humanoid robot collects audio from the environment in real time. If no voice data from the user is detected within a long period (the first time period), it is determined to be in an idle scenario. If the humanoid robot detects voice data from the user within a long period (the first time period), it is determined to be in a dialogue scenario. The first time period is a preset value, which is longer than the duration of the first voice data to be output. For example, the duration of the first voice data to be output can be 5 seconds or 10 seconds, and the duration of the first time period can be 5 minutes or 10 minutes, etc.

[0039] In other embodiments, the time periods for dialogue scenarios and idle scenarios can also be configured by staff. For example, the humanoid robot is in a dialogue scenario during a second time period (such as 10:00~12:00, 16:00~20:00, etc.); and the humanoid robot is in an idle scenario during a third time period (such as 13:00~15:30).

[0040] The first voice data is the preset duration of voice data output by the humanoid robot to the user at the current moment. In idle scenarios, the first voice data can be empty audio of the preset duration. In dialogue scenarios, if the humanoid robot recognizes the user's voice data within the preset duration (e.g., "Hello A, how do I get to shop xx from my current location?"), it converts the recognized user's voice data into text and inputs the text into a large voice model (machine learning model) to obtain the response text (e.g., "Go straight for 200m from the current location, then turn left, and then go forward for another 300m to see shop xx"). After obtaining the response text, it converts the response text into voice data based on text-to-speech (TTS) technology. This voice data is the first voice data. If the user's voice data is not recognized within the preset duration, the first voice data can also be empty audio of the preset duration.

[0041] The preset duration can be configured by staff; for example, the preset duration can be 5s, 8s, or 10s. It should be understood that if the duration of the voice data converted from the reply text is less than the preset duration, the remaining duration will be padded with 0; if the duration of the voice data converted from the reply text is greater than the preset duration, it will be output in batches.

[0042] Both Gaussian noise and the time step are inputs to the subsequent pose generation model. Gaussian noise is a type of random noise that follows a Gaussian distribution (normal distribution), and the time step... is a positive integer, corresponding to the number of denoising steps for randomly sampled Gaussian noise.

[0043] Specifically, the pose generation model is a trained diffusion model. A diffusion model is a deep learning method based on probabilistic generative modeling. Its core idea is to progressively add noise to the data, transforming it into random noise (usually Gaussian white noise), and then gradually denoise it through an inverse process, thereby generating new data samples. Typically, the input is noise. and time step The output is the noise corresponding to the previous time step. Therefore, for T The diffusion model of the step is applied by randomly generating Gaussian noise. Through diffusion model T After denoising, the action sequence without noise can be obtained. .

[0044] The diffusion model, through multi-level denoising, retains more motion features, making the movements closer to real human motion patterns. Moreover, the generation process is guided by the randomness of noise, which can more comprehensively cover the natural variations of the movements, ensuring higher naturalness and fluency of the generated movements. At the same time, since the inverse process starts from random noise, different but reasonable movements can be generated under the same conditions, ensuring the diversity of generated movements.

[0045] Step S102: Determine the first Mel spectrum and the first deep semantic feature based on the first speech data.

[0046] The first deep semantic feature is the first speech data processed by the self-supervised speech representation learning (Hidden unit BERT, HuBERT) model. Specifically, the first speech data is input into the trained HuBERT model, and the output of the trained HuBERT model is the first deep semantic (HuBERT) feature.

[0047] Specifically, the HubERT model is a self-supervised speech representation learning model based on the Transformer architecture. HubERT performs self-supervised speech representation learning through masking predictions of hidden units. It utilizes offline clustering steps to provide aligned target labels for a BERT-like prediction loss. Features are first extracted from unlabeled speech data, and then clustering is used to classify it into different categories. For example, it uses the same strategy as SpanBERT and wav2vec 2.0 to generate masks, applies prediction loss to the masked regions, and forces the model to learn acoustic and language models from continuous inputs to obtain a representation of the sound.

[0048] The first Mel spectrum is the Mel spectrum (basic acoustic feature) corresponding to the first speech data. Mel spectrum is a method of converting the spectrum of an audio signal into a Mel-scale representation, primarily used in speech and audio signal processing. The principle of Mel spectrum is to perform a nonlinear transformation of frequencies using a Mel filter bank to better reflect the human auditory system's perception of frequencies. The calculation of Mel spectrum typically involves Fast Fourier Transform (FFT) and overlapping windowing of frequencies; the resulting Mel spectrum can be used for feature extraction and pattern recognition. The method for calculating the first Mel spectrum from the first speech data is a conventional approach in this field and will not be described in detail here.

[0049] Step S103: Input the first Mel spectrum, the first deep semantic features, the first special action marker, Gaussian noise and time step into the pose generation model, and determine the first action data based on the output of the pose generation model.

[0050] Specifically, the output of the pose generation model is the first motion data, which describes the limb movement information of the humanoid robot when outputting the first voice data. The first motion data may include information such as the angle (or coordinates), speed, torque and torque of the humanoid robot's joints at each moment corresponding to the first voice data.

[0051] Mel spectrograms offer better robustness to noise and environmental changes, enabling pose generation models to maintain stable performance across diverse audio environments. Deep semantic features, on the other hand, allow pose generation models to learn the general semantic information behind speech, rather than being limited to the surface features of specific speech samples. Therefore, compared to directly inputting the first speech data, transforming the first speech data into first Mel spectrograms and first deep semantic features before inputting allows pose generation models to generate more accurate and reasonable actions based on Mel spectrograms and deep semantic features, thus improving generalization ability.

[0052] Furthermore, Mel spectrometry can compress the original linear spectrum into a smaller dimensional space, reducing the amount of data and thus lowering the computational load of the model, improving operational efficiency, and enabling faster action generation. Deep semantic features can provide richer information sources for pose generation. Based on semantic understanding, the model can combine different Mel spectrometry features to generate various actions that conform to semantics and speech rhythm, meeting diverse needs.

[0053] Step S104: When outputting the first voice data, the humanoid robot is simultaneously controlled to generate the gesture action represented by the first motion data.

[0054] Specifically, after determining the first motion data, while controlling the humanoid robot to output the first voice data, the humanoid robot's limbs are controlled based on the first motion data, so that the humanoid robot synchronously outputs gestures corresponding to the first motion data.

[0055] The humanoid robot pose generation method provided in this embodiment first processes the first speech data to obtain the first Mel spectrum and the first deep semantic features after acquiring the first speech data to be output, the first special action marker, Gaussian noise and the time step. Then, the first Mel spectrum, the first deep semantic features, the first special action marker, the Gaussian noise and the time step are input into the pose generation model. The first action data is determined according to the output of the pose generation model. Finally, when outputting the first speech data, the humanoid robot is controlled to synchronously output gesture actions based on the first action data.

[0056] The pose generation model in this embodiment is a pre-trained diffusion model. Due to the randomness of the initial noise in the model, the randomness and diversity of generated actions can be improved, making the humanoid robot more human-like and enhancing the naturalness and intimacy of human-computer interaction. Furthermore, adding a first special action marker to the input enables the output of special actions even when the first speech data is empty audio, ensuring that gesture generation maintains reasonable performance even without speech output. In addition, converting the first speech data into first Mel spectrum and first deep semantic features before input not only enables the pose generation model to generate reasonable actions more accurately and improves generalization ability, but also reduces the model's computational load and improves action generation efficiency.

[0057] The following description, in conjunction with the accompanying diagrams, explains the process of a humanoid robot outputting hand gestures in both idle and dialogue scenarios.

[0058] like Figure 2 As shown, in an idle scenario, the humanoid robot collects speech data from the environment for a preset duration, obtaining a preset duration of empty audio. This empty audio is then converted into Mel spectrum and deep semantic features. Simultaneously, the humanoid robot generates special action markers based on preset frequencies of specific actions. Then, the Mel spectrum, deep semantic features, and randomly generated Gaussian noise are combined... Preset time steps T The model generates a posture by inputting special motion markers. Based on the output of the posture generation model, motion data can be obtained, and finally, the humanoid robot is controlled to output a gesture for a preset duration.

[0059] like Figure 3 As shown, in a dialogue scenario, the humanoid robot collects speech data from the environment for a preset duration, obtaining user speech data (or empty audio) for a preset duration. Based on the collected user speech data (or empty audio), a TTS audio (or empty audio) of a preset duration can be obtained. This TTS audio (or empty audio) is then converted into Mel spectrograms and deep semantic features. Simultaneously, the humanoid robot generates special action markers based on a preset speaking style. Then, the Mel spectrograms, deep semantic features, and randomly generated Gaussian noise are combined... Preset time steps T The model generates a posture by inputting special action markers. Based on the output of the posture generation model, action data can be obtained. Finally, the humanoid robot is controlled to synchronously output TTS audio (or empty audio) of a preset duration and the corresponding gesture actions of the action data.

[0060] In this embodiment, another pose generation method for a humanoid robot is also provided, which can be used for the aforementioned humanoid robot. Figure 4 This is a flowchart illustrating another method for generating poses of a humanoid robot according to an embodiment of the present invention, as shown below. Figure 4As shown, the method includes the following steps: Step S401: Construct a pose generation model based on the sample database.

[0061] The sample database includes multiple training samples, which include second speech data, second action data that follows the changes in the second speech data, and second special action markers. The duration of the second speech data is the same as that of the first speech data.

[0062] Specifically, multiple motion capture actors can record motion data based on TTS voice, with each motion capture actor representing a style. Then, based on the TTS voice, the motion data corresponding to the TTS voice, and the motion capture actor corresponding to the TTS voice, multiple training samples can be obtained.

[0063] Specifically, step S401 includes: Step S4011: Extract target training samples from the sample database.

[0064] The target training sample is one of multiple training samples, which includes target speech data, target action data that changes with the target speech data, and target special action markers.

[0065] Specifically, a training sample can be randomly selected from the sample database as the target training sample. In this case, the second speech data in the selected training sample is the target speech data, the second action data corresponding to the second speech data in the selected training sample is the target motion data, and the second special action marker in the selected training sample is the target special action marker.

[0066] Step S4012: Determine the target Mel spectrum and target deep semantic features based on the target speech data.

[0067] Specifically, after obtaining the target speech data, the target speech data is input into the trained HuBERT model, and the output of the trained HuBERT model is the target deep semantic (HuBERT) feature. The calculation method for the target speech data to the target Mel spectrum is the same as the calculation method for the first speech data to the first Mel spectrum, and will not be explained here.

[0068] Step S4013: Add noise to the target motion data according to the time step to obtain target noise data.

[0069] Specifically, the diffusion model includes a forward process and a reverse process. The forward process involves gradually adding noise to the original data, while the reverse process is the denoising process. For example, if the time step is T, after determining the target motion data, the target motion data after adding T steps of noise is the target noise data. The process of adding noise to the target motion data is a conventional method of adding noise in this field and will not be described in detail here.

[0070] Step S4014: Train the diffusion model based on the target Mel spectrum, target depth semantic features, target noise data, time step, target special action markers, and target action data to obtain the pose generation model.

[0071] Specifically, after determining the target Mel spectrum, target depth semantic features, target noise data, time step, target special action markers, and target action data, the target Mel spectrum, target depth semantic features, target noise data, time step, and target special action markers are input into the diffusion model. The diffusion model denoises the target noise data to obtain predicted action data. Then, the loss between the predicted action data and the target action data is calculated. Based on the calculated loss function, optimization algorithms, such as stochastic gradient descent (SGD) or its variants Adagrad and Adadelta, are used to update the parameters of the diffusion model. After the update, target training samples are reacquired, and the above steps are repeated, continuously adding noise to the target action data, then allowing the updated diffusion model to denoise and predict, calculating the loss, and optimizing the model parameters until the loss function converges, i.e., the difference between the prediction result of the updated diffusion model and the original target action data no longer decreases significantly. At this point, training is complete, and the updated diffusion model is determined as the pose generation model.

[0072] The loss function can be the mean squared error (MSE).

[0073] For example, the process of building a pose generation model can be as follows: Figure 5 As shown, firstly, motion data recorded by the motion capture actor based on TTS speech is acquired, where the TTS speech and motion data are time-aligned. Then, the TTS speech is sliced ​​into fixed-length segments to obtain multiple training samples. Each training sample includes speech data of length L, motion data time-aligned with the speech data of length L, and the style of the speech data (special motion markers). Next, a training sample (i.e., the target training sample) is randomly sampled from the multiple training samples, and then the target motion data is processed using the first time step. t The noise addition process of 1 yields the first noise data. x t1 and performing a second time step on the target motion data. tThe noise addition process of step 2 yields the second noise data. x t2 Simultaneously, the target speech data is processed to obtain the target Mel spectrum and target deep semantic features.

[0074] In this context, the aforementioned time step is greater than the first time step, the first time step is greater than the second time step, and the difference between the first and second time steps is a preset time step. For example, the time step can be 50, 100, or 500, the first time step can be 30 or 20, and the second time step can be 5 or 10.

[0075] After obtaining the first noise data x t1 Second noise data x t2 After the target Mel spectrum and target depth semantic features, the first noise data is... x t1 The target Mel spectrum and target depth semantic features are input into the diffusion model, and the diffusion model is applied to the first noise data. x t1 After performing denoising processing at a preset time step, the predicted noise data is output. Then, the predicted noise data and the second noise data are calculated. x t2 The loss between the two is calculated, and then the parameters of the diffusion model are updated using an optimization algorithm based on the calculated loss. After the update, the target training samples are reacquired, and the above steps are repeated until the loss converges.

[0076] Step S402: Obtain the first speech data to be output, the first special action marker, Gaussian noise, and the time step.

[0077] Please see details Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0078] Step S403: Determine the first Mel spectrum and the first deep semantic feature based on the first speech data.

[0079] Please see details Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0080] Step S404: Input the first Mel spectrum, the first deep semantic features, the first special action marker, Gaussian noise and time step into the pose generation model, and determine the first action data based on the output of the pose generation model.

[0081] Please see details Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0082] Step S405: When outputting the first voice data, the humanoid robot is simultaneously controlled to generate the gesture action represented by the first motion data.

[0083] Please see details Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0084] In this embodiment, before inputting the first Mel spectrum, the first deep semantic features, the first special action marker, Gaussian noise, and the time step into the pose generation model, a pose generation model is constructed using a sample database. The sample database contains diverse training samples, which allows the diffusion model to capture the mapping relationship between the pose and inputs such as the Mel spectrum, semantic features, and special action markers, thereby improving the accuracy of the generated pose.

[0085] For example, such as Figure 6 As shown, the diffusion model can include an input layer, a hidden layer, and an output layer. The hidden layer adopts the Transformer network architecture and includes layer normalization, a multilayer perceptron (MLP), a self-attention layer, and a stylization layer.

[0086] Specifically, the Mel spectrum interacts explicitly with the time steps through a Transformer encoder, enabling it to focus on the conditional features of different time steps (i.e., different noise levels) (e.g., focusing on global rhythm in the early high-noise stage and on detail synchronization in the later low-noise stage), resulting in a spectrum of size L. A matrix of size L with dimensions of 256. HuBERT features are passed through a linear projection layer to obtain a matrix of size L. A matrix of size 128. Concatenating the two matrices along the second dimension yields a matrix of size L. The input layer of the 384 matrix input diffusion model. Here, L represents the preset duration mentioned above. Linear projection layers are a common method for deep neural networks to process input features, aiming to map them into a unified latent space while maintaining their continuity.

[0087] Noise (target noise data, Gaussian noise, or primary noise data) can be L A matrix of size 70, after passing through a linear projection layer, yields a matrix of size L. The input layer of the diffusion model is a 512 matrix, where 70 represents the number of free joints of the humanoid robot. The time step is a positive integer, corresponding to the number of noise addition steps mentioned above. This is passed through a linear projection layer to obtain a 512-length vector input layer for the diffusion model. Special action markers are added to the time step features to guide the model to perform special actions or control the style of actions generated by the diffusion model.

[0088] The Transformer network architecture consists of N Transformer blocks. In one iteration, the audio part and the time step part of the input are fixed (i.e., the two parts of the input are the same for each block), while the noise part is calculated by a series of layers within the block and becomes the output of the block, and at the same time, it becomes the input of the next block, until it passes through N blocks and becomes the final output of the diffusion model.

[0089] Figure 6 The "©" symbol in the diagram represents concatenation, specifically concatenating audio features with noise. Both features have a time axis and are corresponding in time, thus preserving consistency in the time dimension while concatenating along the feature dimension. This allows the hidden layer to process information from both parts uniformly. Layer normalization is a technique used in neural networks to accelerate training and improve model stability. Its core idea is to standardize the intermediate results of a sample.

[0090] Figure 6 The summation in the diagram represents the summation used to implement residual connections. By adding the intermediate results from several layers ago to the current intermediate result, the model can learn the difference between the two, making it easier to converge compared to directly learning a function. The self-attention layer enables the model to dynamically focus on information at different positions in the input sequence, thereby better capturing long-distance dependencies and contextual semantics. In this invention, this layer allows the model to learn coherent actions within a segment.

[0091] In the diffusion model for generating actions, the stylization layer typically considers "correspondence with audio" as a form of stylization. This layer projects the conditional inputs (i.e., audio, time steps, and special action markers) onto the parameters of an affine transformation, and then applies this transformation to the noise input, adjusting the noise distribution to achieve stylization (because different styles have different feature distributions). The repeated use of summation, layer normalization, MLP, and stylization layers increases the network depth, allowing it to learn more content. The output layer is the output of the Transformer block, which is also the output of the entire diffusion model.

[0092] In some implementations, the pose generation method for the humanoid robot further includes steps a1 and a2 before extracting target training samples from the sample database: Step a1: Obtain at least one third voice data and the third action data corresponding to the third voice data.

[0093] The duration of the third voice data is longer than that of the second voice data.

[0094] Specifically, the third voice data can be TTS voice, which can be determined through the humanoid robot's historical TTS voice recordings, and the third motion data can be motion data recorded by a motion capture actor based on the TTS voice. The third motion data can include multiple motions.

[0095] Step a2 involves slicing at least one third speech data to obtain multiple training samples.

[0096] Specifically, by sliding a data window of a preset duration, each third speech data point can be sliced ​​to divide each third speech data point into at least one second speech data point of a preset duration, thereby obtaining multiple training samples.

[0097] In this embodiment, a data window of preset duration divides the third speech data into segments of fixed duration, which can improve the stability of the motion data generated by the diffusion model.

[0098] This invention addresses two humanoid robot interaction scenarios: idle and dialogue. The training processes for the idle and dialogue scenarios differ slightly, and will be described in detail below with reference to the accompanying drawings.

[0099] Specifically, in idle scenarios, the humanoid robot is required to stand naturally with slight swaying, interspersed with special actions such as waving, giving a thumbs up, moving its fingers, and looking down at its hands. This invention introduces special action numbers (i.e., special action tags) as additional conditional information for the diffusion model to control the generation of corresponding preset actions by the diffusion model.

[0100] In idle scenarios, in order to increase the probability of the pose generation model making special movements and ensure the continuity of the movements, the pose generation method of the humanoid robot also includes: setting the weight of the first training sample to a first preset value, setting the weight of the second training sample to a second preset value, and setting the weight of the third training sample to a third preset value before extracting target training samples from the sample database.

[0101] The first training sample is the training sample corresponding to the second action data including the start phase of the action, the second training sample is the training sample corresponding to the second action data including the hold phase of the action, and the third training sample is the training sample corresponding to the second action data including the end phase of the action.

[0102] The first preset value is greater than the second preset value, and the second preset value is greater than the third preset value. For example, the first preset value can be any value between 10 and 100, such as 10, 20, 40, 50, 60, 80, or 100. The second preset value can be greater than or equal to 1 and less than 10, such as 1, 2, 5, 6, or 9. The third preset value can be greater than or equal to 0 and less than 1, such as 0, 0.1, 0.5, 0.7, or 0.9.

[0103] Specifically, a diagram of the third action data can be shown as follows: Figure 7 As shown, Figure 7 The horizontal axis represents the time axis, and the vertical axis represents the arm height. The arm height refers to the rotation angle of a certain joint of the humanoid robot's arm; the larger the value, the higher the arm is raised. Figure 7 (This is just an example, not actual values). A flat segment indicates that the arm is basically still (i.e., standing naturally), and a raised segment indicates that the arm is raised (i.e., a special movement). The movement label on the timeline corresponds to the arm height; 0 indicates almost no movement, and 1 indicates movement number 1 (the number is the special movement marker mentioned earlier). For example... Figure 7 As shown, the execution of a special action includes the action start phase, the action hold phase, and the action end phase.

[0104] During training, to increase the probability of the diffusion model performing the corresponding action, the sampling probability of training samples at the beginning of the action was increased by setting a first preset value. Meanwhile, since the action at the end of the action is usually the process of lowering the arm (corresponding to a decrease in arm height), this data can interfere with the action distribution during model training. This can cause the model to easily generate actions where the arm is raised and immediately lowered when generating continuous, coherent movements. Therefore, a third preset value was set to reduce (or even discard) this data.

[0105] For example, if the first preset value is 10 to 100 and the second preset value is 1, then the sampling probability of the training samples in the action start phase can be 10 to 100 times that of the sampling probability of the training samples in the action hold phase.

[0106] Specifically, in dialogue scenarios, the humanoid robot is required to perform actions that match its rhythm and content while outputting speech, and to maintain a natural standing posture when not speaking. To ensure that the same model can handle different actions in both dialogue and idle states, and to optimize the synchronization between speaking actions and speech rhythm, in some embodiments, the humanoid robot pose generation method further includes steps b1 and b2 before training the diffusion model: Step b1: Determine whether the target Mel spectrum includes 0.

[0107] Specifically, after obtaining the target Mel spectrum, each element in the Mel spectrum matrix can be checked to determine if there is a value equal to 0.

[0108] Step b2: When the target Mel spectrum includes 0, map the regions where the target Mel spectrum is 0 and the parts of the target deep semantic features corresponding to the regions where the target Mel spectrum is 0 to learnable parameters to obtain the updated target Mel spectrum and the updated target deep semantic features.

[0109] Learnable parameters are model parameters that can learn good representations of empty audio in the encoding space. These can replace Mel-spectral and deep semantic features of empty audio, improving the performance of pose generation models in generating action data. Learnable parameters can capture additional information in the empty audio and associate it with corresponding actions through gradient backpropagation in deep learning.

[0110] Specifically, when the target Mel spectrum includes 0, a parameterized function (such as a neural network layer) can be used to map the region where the target Mel spectrum is 0 to a new feature value (i.e., a learnable parameter) to obtain the updated target Mel spectrum. In addition, the part of the target deep semantic features corresponding to the region where the target Mel spectrum is 0 can be mapped to learnable parameters to obtain the updated target deep semantic features.

[0111] For example, such as Figure 8 As shown, the blank regions in the three Mel spectra are represented as regions of 0. In this case, the blank regions in the three Mel spectra are replaced by learnable parameters.

[0112] In one example, such as Figure 9 As shown, after obtaining the target Mel spectrum, it is first determined whether the target Mel spectrum includes 0. If it includes 0, the corresponding parts of the target Mel spectrum and deep semantic features are replaced with learnable parameters before being input into the diffusion model; if it does not include 0, the target Mel spectrum and deep semantic features are directly input into the diffusion model.

[0113] At this point, step S4014 specifically involves training the diffusion model based on the updated target Mel spectrum, updated target depth semantic features, target noise data, time step, target special action markers, and target action data.

[0114] Specifically, for frames with a Mel spectrum of 0 (including empty audio in idle scenes and pauses in TTS speech), the audio encoding is mapped to learnable parameters, and the best empty audio representation is learned in the encoding space, which enhances the model's ability to distinguish between speaking and idle states.

[0115] In this embodiment, the process of replacing the region with a Mel spectrum of 0 with a learnable parameter is called frame-by-frame mode. In frame-by-frame mode, the diffusion model can fully associate empty audio with non-speaking actions, so that even if only a part of a segment of length L is empty audio, the diffusion model can still generate non-speaking actions accordingly.

[0116] In other implementations, the pose generation method for the humanoid robot further includes steps c1 and c2 before training the diffusion model: Step c1: Determine whether the target Mel spectrum is all 0.

[0117] Specifically, after obtaining the target Mel spectrum, each element in the Mel spectrum matrix can be checked to determine whether all elements are 0.

[0118] Step c2: When the target Mel spectrum is all 0, map the target Mel spectrum and the target deep semantic features into learnable parameters to obtain the first model parameters and the second model parameters.

[0119] Specifically, when the target Mel spectrum is all 0, the target Mel spectrum can be mapped to new feature values ​​(i.e., learnable parameters) through a parameterized function (such as a neural network layer) to obtain the first model parameters, and the deep semantic features can be mapped to learnable parameters to obtain the second model parameters.

[0120] For example, such as Figure 10 As shown, the blank regions in the three Mel spectra represent regions of 0. At this time, only the second Mel spectrum is all 0. In this embodiment, it is only necessary to map the second Mel spectrum to learnable parameters.

[0121] In cases where the target Mel spectrum is not entirely zero, the target Mel spectrum and deep semantic features are directly input into the diffusion model.

[0122] At this point, step S4014 specifically involves training the diffusion model based on the first model parameters, the second model parameters, the target noise data, the time step, the target special action markers, and the target action data.

[0123] In this embodiment, the process of replacing the Mel spectrum with all zeros with learnable parameters is called the sample pattern. In the sample pattern, the diffusion model can learn more coherent non-speaking state actions.

[0124] This embodiment also provides a posture generation device for a humanoid robot, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0125] This embodiment provides a posture generation device for a humanoid robot, such as... Figure 11 As shown, it includes: The acquisition module 1101 is used to acquire the first speech data to be output, the first special action marker, Gaussian noise and time step, wherein the first special action marker is used to characterize the style of the generated gesture action or the preset special action; The feature determination module 1102 is used to determine a first Mel spectrum and a first deep semantic feature based on the first speech data, wherein the first deep semantic feature is the first speech data processed by a self-supervised speech representation learning model; The action determination module 1103 is used to input the first Mel spectrum, the first deep semantic features, the first special action marker, Gaussian noise and time step into the pose generation model, and determine the first action data based on the output of the pose generation model; The output module 1104 is used to synchronously control the humanoid robot to generate the gesture action represented by the first motion data when outputting the first voice data.

[0126] In some alternative embodiments, the apparatus further includes: The building module is used to build a pose generation model based on a sample database. The sample database includes multiple training samples, which include second speech data, second action data that follows the changes in the second speech data, and second special action markers. The duration of the second speech data is the same as that of the first speech data.

[0127] In some alternative implementations, the building module includes: The extraction unit is used to extract target training samples from the sample database. The target training sample is one of multiple training samples. The target training sample includes target speech data, target action data that changes with the target speech data, and target special action markers. The first determining unit is used to determine the target Mel spectrum and target deep semantic features based on the target speech data; The noise-adding unit is used to add noise to the target motion data according to the time step to obtain target noise data; The training unit is used to train the diffusion model based on the target Mel spectrum, target depth semantic features, target noise data, time steps, target special action markers, and target action data to obtain the pose generation model.

[0128] In some alternative embodiments, the apparatus further includes: The first determining module is used to determine whether the target Mel spectrum includes 0; The first mapping module is used to map the region where the target Mel spectrum is 0 and the part of the target deep semantic features corresponding to the region where the target Mel spectrum is 0 into learnable parameters when the target Mel spectrum includes 0, so as to obtain the updated target Mel spectrum and the updated target deep semantic features. The training unit includes: a first sub-training unit, used to train the diffusion model based on the updated target Mel spectrum, updated target depth semantic features, target noise data, time steps, target special action markers, and target action data.

[0129] In some alternative embodiments, the apparatus further includes: The second determining module is used to determine whether the target Mel spectrum is all 0; The second mapping module is used to map the target Mel spectrum and the target deep semantic features into learnable parameters when the target Mel spectrum is all 0, so as to obtain the first model parameters and the second model parameters. The training unit includes a second sub-training unit, which is used to train the diffusion model based on the first model parameters, the second model parameters, the target noise data, the time step, the target special action markers, and the target action data.

[0130] In some alternative embodiments, the apparatus further includes: The data acquisition module is used to acquire at least one third voice data and the third action data corresponding to the third voice data, wherein the duration of the third voice data is greater than the duration of the second voice data. The slicing module is used to slice at least one third speech data to obtain multiple training samples.

[0131] In some optional embodiments, the third motion data includes a motion start phase, a motion hold phase, and a motion end phase, and the device further includes: The processing module is used to set the weight of the first training sample to a first preset value, set the weight of the second training sample to a second preset value, and set the weight of the third training sample to a third preset value. The first training sample is the training sample corresponding to the second action data including the action start phase, the second training sample is the training sample corresponding to the second action data including the action hold phase, and the third training sample is the training sample corresponding to the second action data including the action end phase. The first preset value is greater than the second preset value, and the second preset value is greater than the third preset value.

[0132] In some optional implementations, the first voice data is empty audio, and the first special action marker is determined according to the frequency of a preset special action.

[0133] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0134] In this embodiment, the humanoid robot's posture generation device is presented in the form of a functional unit. Here, a unit refers to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0135] This invention also provides a humanoid robot, such as... Figure 12 As shown, the humanoid robot includes one or more processors 1210, a memory 1220, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the humanoid robot, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple humanoid robots can be connected, each providing some of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). Figure 12 Take the 1210 processor as an example.

[0136] Processor 1210 may be a central processing unit, a network processor, or a combination thereof. Processor 1210 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPRS), or any combination thereof.

[0137] The memory 1220 stores instructions executable by at least one processor 1210 to cause the at least one processor 1210 to perform the method shown in the above embodiments.

[0138] The memory 1220 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the humanoid robot. Furthermore, the memory 1220 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 1220 may optionally include memory remotely located relative to the processor 10, and these remote memories can be connected to the humanoid robot via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0139] The memory 1220 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 1220 may also include combinations of the above types of memory.

[0140] The humanoid robot also includes an input device 1230 and an output device 1240. The processor 1210, memory 1220, input device 1230, and output device 1240 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0141] Input device 1230 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the humanoid robot, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1240 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0142] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a humanoid robot, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that the humanoid robot, processor, microprocessor controller, or programmable hardware includes storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the humanoid robot, processor, or hardware, the methods shown in the above embodiments are implemented.

[0143] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a humanoid robot, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the humanoid robot. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Accordingly, the ways in which computer program instructions are executed by a humanoid robot include, but are not limited to: the humanoid robot directly executing the instructions, or the humanoid robot compiling the instructions and then executing the corresponding compiled program, or the humanoid robot reading and executing the instructions, or the humanoid robot reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the humanoid robot.

[0144] In the description of this specification, the references to terms such as "this embodiment," "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0146] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the present invention.

Claims

1. A method for generating poses of a humanoid robot, characterized in that, The method includes: The first speech data to be output, the first special action marker, Gaussian noise and time step are obtained, wherein the first special action marker is used to characterize the style of the generated gesture or the preset special action; Based on the first speech data, a first Mel spectrum and a first deep semantic feature are determined, wherein the first deep semantic feature is the first speech data processed by a self-supervised speech representation learning model; A pose generation model is constructed based on a sample database, wherein the sample database includes multiple training samples, and the training samples include second speech data, second action data that follows the changes of the second speech data, and second special action markers, wherein the duration of the second speech data is the same as the duration of the first speech data. The first Mel spectrum, the first deep semantic feature, the first special action marker, the Gaussian noise, and the time step are input into the pose generation model, and the first action data is determined based on the output of the pose generation model, wherein the pose generation model is a diffusion model that has been trained. While outputting the first voice data, the humanoid robot is simultaneously controlled to generate the gesture action represented by the first motion data; The step of constructing a pose generation model based on a sample database includes: extracting target training samples from the sample database, wherein the target training sample is one of a plurality of training samples, and the target training sample includes target speech data, target action data that follows the changes in the target speech data, and target special action markers; determining target Mel spectrum and target deep semantic features based on the target speech data; adding noise to the target action data according to the time step to obtain target noise data; and training the diffusion model based on the target Mel spectrum, the target deep semantic features, the target noise data, the time step, the target special action markers, and the target action data to obtain the pose generation model.

2. The method according to claim 1, characterized in that, Before training the diffusion model, the method further includes: Determine whether the target Mel spectrum includes 0; When the target Mel spectrum includes 0, the region where the target Mel spectrum is 0 and the part of the target deep semantic feature corresponding to the region where the target Mel spectrum is 0 are mapped to learnable parameters to obtain the updated target Mel spectrum and the updated target deep semantic feature; The step of training the diffusion model based on the target Mel spectrum, the target deep semantic features, the target noise data, the time step, the target special action markers, and the target action data includes: The diffusion model is trained based on the updated target Mel spectrum, the updated target deep semantic features, the target noise data, the time step, the target special action markers, and the target action data.

3. The method according to claim 1, characterized in that, Before training the diffusion model, the method further includes: Determine whether the target Mel spectrum is all 0; When the target Mel spectrum is all 0, the target Mel spectrum and the target deep semantic features are mapped to learnable parameters to obtain the first model parameters and the second model parameters; The step of training the diffusion model based on the target Mel spectrum, the target deep semantic features, the target noise data, the time step, the target special action markers, and the target action data includes: The diffusion model is trained based on the first model parameters, the second model parameters, the target noise data, the time step, the target special action markers, and the target action data.

4. The method according to any one of claims 1 to 3, characterized in that, Before extracting target training samples from the sample database, the method further includes: Acquire at least one third voice data and third action data corresponding to the third voice data, wherein the duration of the third voice data is greater than the duration of the second voice data; The at least one third speech data is sliced ​​to obtain the plurality of training samples.

5. The method according to claim 4, characterized in that, The third motion data includes the motion start phase, the motion hold phase, and the motion end phase. Before extracting target training samples from the sample database, the method further includes: The weights of the first training sample are set to a first preset value, the weights of the second training sample are set to a second preset value, and the weights of the third training sample are set to a third preset value. The first training sample is a training sample corresponding to the second action data including the action start phase, the second training sample is a training sample corresponding to the second action data including the action hold phase, and the third training sample is a training sample corresponding to the second action data including the action end phase. The first preset value is greater than the second preset value, and the second preset value is greater than the third preset value.

6. The method according to any one of claims 1 to 3, characterized in that, The first voice data is empty audio, and the first special action marker is determined according to the frequency of a preset special action.

7. A posture generation device for a humanoid robot, characterized in that, The device includes: The acquisition module is used to acquire the first speech data to be output, the first special action marker, Gaussian noise and time step, wherein the first special action marker is used to characterize the style of the generated gesture or the preset special action; The feature determination module is used to determine a first Mel spectrum and a first deep semantic feature based on the first speech data, wherein the first deep semantic feature is the first speech data processed by a self-supervised speech representation learning model; A construction module is used to construct a pose generation model based on a sample database, wherein the sample database includes multiple training samples, the training samples include second speech data, second action data that follows the changes of the second speech data, and second special action markers, and the duration of the second speech data is the same as the duration of the first speech data; An action determination module is used to input the first Mel spectrum, the first deep semantic features, the first special action marker, the Gaussian noise, and the time step into the pose generation model, and determine the first action data based on the output of the pose generation model, wherein the pose generation model is a trained diffusion model; The output module is used to synchronously control the humanoid robot to generate the gesture action represented by the first action data when outputting the first voice data; The step of constructing a pose generation model based on a sample database includes: extracting target training samples from the sample database, wherein the target training sample is one of a plurality of training samples, and the target training sample includes target speech data, target action data that follows the changes in the target speech data, and target special action markers; determining target Mel spectrum and target deep semantic features based on the target speech data; adding noise to the target action data according to the time step to obtain target noise data; and training the diffusion model based on the target Mel spectrum, the target deep semantic features, the target noise data, the time step, the target special action markers, and the target action data to obtain the pose generation model.

8. A humanoid robot, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the posture generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the humanoid robot to perform the posture generation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a humanoid robot to perform the pose generation method according to any one of claims 1 to 6.

Images