Method, equipment, medium and program product for generating robot limb movement

Abstract

The invention aims to provide a method and equipment for generating robot limb actions, a medium and a program product, and the method comprises the following steps: based on interaction information and cue word information, generating a reply text and one or more action scripts corresponding to the interaction information through a large language model; determining an action track corresponding to each action script from an action library; determining voice information and timestamp information corresponding to the reply text by using a text-to-voice algorithm; the method comprises the steps of receiving voice information sent by a user, and then determining a target limb action matched with the voice information, the target limb action comprising one or more action tracks and trigger timestamp information corresponding to each action track. The action generation efficiency is improved, and the synchronism of robot actions and voice reply playing is ensured.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a technology for generating robot limb movements. Background Technology

[0002] In humanoid robot interactions, body language is a key feature that distinguishes it from other human-computer interaction scenarios. In the past, most robot human-computer interaction feedback strategies were based on pre-programmed models, meaning that all responses—voice, actions, facial expressions, and their triggering sequence—had to be manually set and arranged by developers before the interaction occurred. This static interaction feedback strategy not only generates limited content but also struggles to adapt to open and dynamic dialogue contexts.

[0003] Currently, with the penetration and development of deep learning technology in the field of robotics, some methods have emerged that can automatically generate full-body coordinated and semantically rich motion trajectories based on natural language descriptions. These methods first determine the response text using a deep learning model, then parse the response text into a semantic description of the action (such as action category, duration, etc.), outputting a corresponding structured semantic template. A motion generation model then maps the semantic vectors into continuous joint angle trajectories to generate full-body movements. However, the generated movements are difficult to align with the robot's output speech, consume significant computational resources, and have low computational efficiency, often making them unsuitable for real-time interactive control scenarios. Summary of the Invention

[0004] One object of this application is to provide a method, apparatus, medium, and program product for generating robot limb movements.

[0005] According to one aspect of this application, a method for generating robot limb movements is provided, the method comprising:

[0006] Based on the interaction information and prompt word information, a large language model is used to generate the response text corresponding to the interaction information and one or more action scripts. The prompt word information includes several interaction examples, and the action scripts include action tag information.

[0007] Determine the action trajectory corresponding to each action script from the action library;

[0008] Using a text-to-speech algorithm, the audio information corresponding to the reply text is determined, wherein the audio information includes the speech information corresponding to the reply text and the timestamp information corresponding to each character in the reply text;

[0009] Based on the motion trajectory corresponding to each motion script and the audio information, a target limb action matching the voice information is determined, wherein the target limb action includes one or more motion trajectories and trigger timestamp information corresponding to each motion trajectory.

[0010] According to one aspect of this application, a computer device for generating robot limb movements is provided, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of any of the methods described above.

[0011] According to one aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0012] According to one aspect of this application, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of any of the methods described above.

[0013] According to one aspect of this application, an apparatus for generating robot limb movements is provided, the apparatus comprising:

[0014] The module is used to generate response text and one or more action scripts corresponding to the interaction information and prompt word information through a large language model, wherein the prompt word information includes several interaction examples and the action script includes action tag information.

[0015] Modules one and two are used to determine the motion trajectory corresponding to each motion script from the motion library;

[0016] The first and third modules are used to determine the audio information corresponding to the reply text using a text-to-speech algorithm. The audio information includes the speech information corresponding to the reply text and the timestamp information corresponding to each character in the reply text.

[0017] The first four modules are used to determine the target limb movement that matches the voice information based on the movement trajectory corresponding to each action script and the audio information. The target limb movement includes one or more movement trajectories and trigger timestamp information corresponding to each movement trajectory.

[0018] Compared with existing technologies, this application, based on interaction information and prompt word information, generates response text and one or more action scripts corresponding to the interaction information through a large language model. The prompt word information includes several interaction examples, and the action scripts include action tag information. The application determines the action trajectory corresponding to each action script from an action library. Using a text-to-speech algorithm, it determines the audio information corresponding to the response text, including the speech information and timestamp information corresponding to each character in the response text. Based on the action trajectory corresponding to each action script and the audio information, it determines the target limb action matching the speech information, including one or more action trajectories and trigger timestamp information corresponding to each action trajectory. This application employs prompt word engineering, simultaneously generating response text and action scripts through a large language model, improving action generation efficiency. Furthermore, by using a text-to-speech algorithm to output character-level timestamps during audio information synthesis, it precisely controls the triggering timing of action trajectories, ensuring synchronized playback of robot actions and speech responses, guaranteeing the clarity of action semantics, and facilitating accurate user understanding of their meaning. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0020] Figure 1 A flowchart of a method for generating robot limb movements according to an embodiment of this application is shown;

[0021] Figure 2 A flowchart of a method for generating robot limb movements according to an embodiment of this application is shown;

[0022] Figure 3 This is a structural diagram of a device for generating robot limb movements according to an embodiment of the present application;

[0023] Figure 4 Exemplary systems that can be used to implement the various embodiments described in this application are shown.

[0024] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings.

[0026] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), input / output interfaces, network interfaces, and memory.

[0027] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0028] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0029] The devices referred to in this application include, but are not limited to, user equipment, network equipment, or devices composed of user equipment and network equipment integrated through a network. The user equipment includes, but is not limited to, any mobile electronic product capable of human-computer interaction (e.g., via a touchpad), such as smartphones and tablets. These mobile electronic products can use any operating system, such as Android or iOS. The network equipment includes an electronic device capable of automatically performing numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. The network equipment includes, but is not limited to, computers, network hosts, single network servers, multiple network server clusters, or clouds composed of multiple servers. Here, a cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, wireless ad hoc network, etc. Preferably, the device can also be a program running on the user equipment, network device, or a device formed by integrating user equipment and network device, network device, touch terminal, or network device and touch terminal through a network.

[0030] Of course, those skilled in the art should understand that the above-described devices are merely examples, and other existing or future devices that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0031] In the description of this application, "multiple" means two or more, unless otherwise expressly and specifically defined.

[0032] Figure 1The diagram illustrates a method for generating robot limb movements according to an embodiment of this application. The method includes steps S11, S12, S13, and S14. In step S11, device 1 generates a response text and one or more action scripts corresponding to the interaction information using a large language model, based on interaction information and prompt word information. The prompt word information includes several interaction examples, and the action scripts include action tag information. In step S12, device 1 determines the action trajectory corresponding to each action script from an action library. In step S13, device 1 uses a text-to-speech algorithm to determine the audio information corresponding to the response text. The audio information includes the speech information corresponding to the response text and the timestamp information corresponding to each character in the response text. In step S14, device 1 determines a target limb movement matching the speech information based on the action trajectory corresponding to each action script and the audio information. The target limb movement includes one or more action trajectories and the trigger timestamp information corresponding to each action trajectory.

[0033] In some embodiments, device 1 may be a computing platform deployed on a robot that meets the computing power required by the present application.

[0034] In step S11, device 1 generates a response text and one or more action scripts corresponding to the interaction information based on the interaction information and the prompt word information through a large language model. The prompt word information includes several interaction examples, and the action scripts include action tag information.

[0035] In some embodiments, the interaction information is the acquired user's voice input content. In some embodiments, the method further includes step S15 (not shown), determining the interaction information based on a voice activity detection model and an automatic speech recognition model. For example, a voice activity detection (VAD) model is used to detect user voice, and when user voice is detected, an automatic speech recognition (ASR) model is used to recognize the user's voice. When the VAD model detects that the user has finished speaking, the ASR model outputs the user's voice text content, which can be used as interaction information input into the large language model.

[0036] In some embodiments, to enable the large language model to simultaneously output corresponding action scripts while outputting response text corresponding to interactive information, this application uses an in-context learning method to guide the large language model's inference through prompt word engineering. Several "language-action" interaction examples are introduced into the prompt words as references for the model to generate action scripts, allowing the large language model to generate action scripts matching the interactive language without retraining or fine-tuning. Furthermore, the prompt word information enables the large language model to generate response text with embedded action scripts in a single inference, reducing additional computational overhead. Here, the large language model can be selected from, but is not limited to, GPT-4, Gemini 3, Claude 3, or higher series large language models with strong inference capabilities. The interaction examples include response text examples and several corresponding action script examples. The interaction examples can be flexibly updated based on the correspondence between actions and language required in actual applications. The action script examples can correspond one-to-one with words, phrases, or short sentences in the response text examples. Considering the accessibility of actions during real device deployment, comma-separated short sentences are typically used as the basic unit for action triggering. That is, the action script examples correspond one-to-one with the comma-separated clauses in the response text examples. For example, an interaction example could be: Response text example: {clause 1, clause 2, clause 3.}, Action script example: {action script 1, action script 2, action script 3}. In some embodiments, each action script in the one or more action scripts matches one or more characters in the response text. The action script includes action tag information. The action tag information corresponds to pre-recorded action templates in the action library. For example, the determined response text is "Thank you for your help, see you next time." Accordingly, the action script could be "bow" and "wave". In some embodiments, to make the robot output synchronized with the voice output and conform to the context, the action script may also include action duration word count information and / or emotion tag information. For example, for the response text "Thank you for your help, see you next time.", its corresponding action script could be "gesture:bow, duration:6, emotion:happy" and "gesture:wave, duration:4, emotion:happy". In some embodiments, the action script may further include identification information for one or more characters in the response text corresponding to the action script. In some embodiments, to enable the large language model to understand that not all responses require action output, the prompt information may also include interaction examples that do not trigger actions. For example, the interaction example may only contain a response text example, and the corresponding action script example may be marked with "0".

[0037] Those skilled in the art will understand that the above examples are merely illustrative and this embodiment does not limit the scope of the invention.

[0038] In step S12, device 1 determines the motion trajectory corresponding to each motion script from the motion library.

[0039] In some embodiments, the action library includes pre-recorded action templates corresponding to each action tag. Device 1 can query the action templates corresponding to the action tag information in these action scripts from the action library. The queried action templates can be directly output as the action trajectory along with the robot's response text.

[0040] In some embodiments, to improve the appropriateness and richness of robot body movements, the determined action template can be stylized to generate differentiated action trajectories, allowing the robot to express different emotional states or social styles. Specifically, the action script also includes emotion tag information. Step S12 includes: for each action script, querying the action library to determine the corresponding action template based on the action tag information; generating corresponding correction parameters using a trained multilayer perceptron model based on the emotion tag information and the action template; and determining the action trajectory corresponding to the action script based on the correction parameters and the action template. For example, for each action script, querying the action library for the corresponding action template using the action tag information can transform the action template q base (t) is taken as the baseline action, and combined with sentiment label information, the corresponding correction parameter Δq(t)=f is output using a trained multilayer perceptron (MLP) model. MLP (t,e,q base (t)). The final motion trajectory q is determined by combining the correction parameters and the motion template. final (t)=q base(t)+Δq(t). Where e is the sentiment vector of the input MLP determined based on sentiment label information. In some embodiments, the sentiment label information can be encoded to convert it into a vector form suitable for model processing. In some embodiments, one-hot encoding can be used to assign a value of 1 to the vector dimension corresponding to the sentiment label information of the action script, and assign a value of 0 to the other vector dimensions. For example, if the sentiment label set includes sentiment labels such as happy, sad, angry, and calm, and if the sentiment label information corresponding to the action script is happy, the encoded sentiment vector for the sentiment label set [happy, sad, angry, calm] will be [1,0,0,0]. In some embodiments, multiple sentiment hybrid control encodings can also be used to assign values ​​to the sentiment vector. For example, based on multiple sentiment label information participating in the fusion, a value in the range [0.0,1.0] can be assigned to the vector dimension corresponding to the corresponding sentiment label information, which represents the weight ratio of the corresponding sentiment in the hybrid sentiment. For example, for the set of emotion labels [happy, sad, angry, calm], if happiness is the main factor and calmness is the secondary factor, the emotion vector obtained after mixed control encoding is [0.7,0,0,0.3].

[0041] Those skilled in the art should understand that the above examples of sentiment tagging information and encoding methods are merely illustrative and are not intended to limit the scope of this application. Other existing or future sentiment tagging information and encoding methods that are applicable to this application should also be included within the scope of protection of this application and are hereby incorporated by reference.

[0042] In some embodiments, the stylized multilayer perceptron model can be trained based on a manually labeled training dataset (typically containing original motion trajectories, sample sentiment information, and sample motion trajectories adjusted based on the sample sentiment information). The multilayer perceptron model is then deployed on a robot computing platform for stylized motion trajectory adjustment. Specifically, the method further includes: step S16 (not shown), generating corresponding sample correction parameters using the multilayer perceptron model to be trained based on the original motion trajectories and sample sentiment information in the training dataset; determining corresponding corrected motion trajectories based on the sample correction parameters and the original motion trajectories; updating the model parameters based on the corrected motion trajectories and the sample motion trajectories in the training dataset corresponding to the original motion trajectories and sample sentiment information, combined with a corresponding loss function, to obtain the trained multilayer perceptron model.

[0043] In some embodiments, this application introduces a small MLP model containing hundreds of thousands of parameters. The MLP model includes an input layer, one or more hidden layers, and an output layer, with each layer having a linear mapping relationship established between weight matrices and bias vectors. All weight matrices and bias vectors can be initialized before training. During training, based on the original action trajectories and sample sentiment information in the training dataset, the MLP model determines the corresponding sample correction parameters, and combines these parameters with the original action trajectories to determine the corrected action trajectory q. pred This process is similar to the process of determining the motion trajectory in step S12 mentioned above, so it will not be described again, but is included here by reference. Furthermore, the motion trajectory q can be corrected based on this. pred The corresponding sample action trajectory q in the training dataset target Based on the corresponding loss function, loss is calculated. Here, the loss function can be the Mean Squared Error (MSE) loss function. Based on the calculated loss, the partial derivatives (i.e., gradients) of the loss function with respect to the model parameters (weight matrices and bias vectors of each layer) can be calculated using the backpropagation algorithm. Then, the parameters are updated using an appropriate optimizer (e.g., Adam optimizer, RMSprop, or Adagrad), ultimately completing the training to obtain the pre-trained multilayer perceptron model used above.

[0044] In some embodiments, adjustments can be made to the original motion trajectory in dimensions such as amplitude, speed, and force based on preset rules to generate corresponding sample motion trajectories to simulate the body movement expression under the corresponding sample emotional information, thereby constructing a dataset for model training. Specifically, the method further includes: step S17 (not shown), generating corresponding sample motion trajectories based on the sample emotional information and the original motion trajectory using preset emotional style rules, and establishing a corresponding training dataset. For example, the original motion trajectory may be a preset motion template in an action library. The sample emotional information may be manually set possible emotional label information for the original motion trajectory. For example, for the "wave" action, there may be "happy wave" or "sad wave," and correspondingly, the sample emotional information may be happy or sad. Furthermore, based on the preset emotional style rules and the sample emotional information, adjustments can be made to the original motion trajectory in dimensions such as amplitude, speed, and force to obtain a sample motion trajectory that conforms to the emotional expression corresponding to the sample emotional information. The emotional style rules include action adjustment rules corresponding to each emotional label information. For example, "happy" corresponds to increasing the amplitude of the movement; "sad" corresponds to decreasing the amplitude and speed; and "angry" corresponds to increasing the speed and intensity. The sample movement trajectory obtained based on these adjustments, along with the corresponding original movement trajectory and sample emotional information, can be used as a training sample. By repeating the above steps, the required number of samples are generated, thus constructing the corresponding training dataset.

[0045] Those skilled in the art should understand that the above examples of emotional style rules are merely illustrative and are not intended to limit the scope of this application. Other existing or future emotional style rules that are applicable to this application should also be included within the scope of protection of this application and are hereby incorporated by reference.

[0046] In step S13, device 1 uses a text-to-speech algorithm to determine the audio information corresponding to the response text. The audio information includes the speech information corresponding to the response text and the timestamp information corresponding to each character in the response text. For example, traditional text-to-speech (TTS) algorithms (e.g., Tacotron2, FastSpeech2, etc.) can be used to determine the corresponding Mel spectrum and the timestamp information corresponding to each character in the response text based on the response text. Then, the corresponding speech information is generated using a corresponding vocoder based on the Mel spectrum. Alternatively, an end-to-end TTS algorithm (e.g., WaveGrad2, VITS model, etc.) can be used to directly generate the corresponding speech information and the timestamp information corresponding to each character in the response text based on the response text. In some embodiments, the timestamp information corresponding to each character includes the speech start time and speech end time corresponding to that character.

[0047] This application does not limit the execution order of step S12, which determines the action trajectory corresponding to each action script, and step S13, which determines the audio information corresponding to the response text. They can be performed simultaneously, or the action trajectory can be determined first and then the audio information, or the audio information can be determined first and then the action trajectory, etc.

[0048] In step S14, device 1 determines a target limb movement matching the voice information based on the motion trajectory corresponding to each action script and the audio information. The target limb movement includes one or more motion trajectories and trigger timestamp information corresponding to each motion trajectory. For example, based on the correspondence between each action script and one or more characters in the response text, the first character among the one or more characters corresponding to the action script can be determined, and the timestamp information corresponding to that character can be used as the trigger timestamp information for the motion trajectory corresponding to that action script. In some embodiments, the timestamp information corresponding to each character includes the voice start time and voice end time corresponding to that character, and the voice start time corresponding to that character can be used as the trigger timestamp information for the motion trajectory. Thus, a target limb movement synchronized with the voice output can be obtained. In some embodiments, to make the robot's voice and limb movement interaction more natural and in line with the human-computer interaction perception rules, for non-first action scripts, the timestamp t of the first character corresponding to that action script can be preset by a time (e.g., 0.15s) as the trigger timestamp (e.g., t-0.15s) corresponding to that action script.

[0049] For example, refer to Figure 2 This paper demonstrates a process for generating robot response speech and synchronized actions: First, a large language model is used to generate corresponding action scripts and response text based on input prompt words and the user's perceived speech text. Then, based on the action scripts, action library, and response text, MLP and TTS models are used to generate corresponding action trajectories and timestamped speech information, thereby determining the timing of the speech to be played and the action trajectory to be executed. This allows for precise control of the triggering timing of the action trajectory, achieving synchronization between robot actions and speech. Furthermore, through prompt word engineering, the large language model can generate response text embedded with action scripts in a single inference, reducing additional computational overhead and improving computational efficiency.

[0050] In some embodiments, the action script further includes action duration word count information; the method further includes: step S18 (not shown), for each action script corresponding to an action trajectory, if the playback duration of the action trajectory is greater than the duration corresponding to the corresponding action duration word count information, the action trajectory is updated. For example, the duration corresponding to the action duration word count information can be determined based on the timestamp information corresponding to each character. If the playback duration corresponding to the action trajectory is greater than the duration, the playback rate of the action trajectory can be accelerated by linearly scaling the action trajectory, so that its playback duration matches the duration. This facilitates the playback scheduling of the action trajectory.

[0051] In some embodiments, the method further includes: step S19 (not shown), when the voice information is played to the corresponding trigger timestamp information, executing the motion trajectory corresponding to the trigger timestamp information in the target limb movement. For example, the playback progress can be monitored in real time when the voice information starts playing, and it can be determined whether the current playback time is greater than or equal to the corresponding trigger timestamp information and whether the motion trajectory corresponding to the trigger timestamp information has not been triggered. If so, the corresponding joint motor of the robot can be driven to move to execute the motion trajectory; otherwise, monitoring continues. This achieves synchronized playback of voice and corresponding limb movements.

[0052] In some embodiments, step S19 includes: when the voice information is played to the corresponding trigger timestamp information, if there is an action trajectory being executed, the execution of the action trajectory is terminated, and the action trajectory corresponding to the trigger timestamp information in the target limb action is executed. That is, in the case where the previous action has not finished playing and the next action is already waiting to be triggered, in order to ensure the consistency of the interaction rhythm between voice and limb action, the current action can be terminated and the action trajectory to be triggered can be executed directly.

[0053] In some embodiments, step S19 includes: when the voice information is played to the corresponding trigger timestamp information, if there is an ongoing action trajectory, perform a corresponding first scheduling operation based on the interruptibility attribute of the ongoing action trajectory. Further, when the previous action has not finished playing and the next action is about to be triggered, dynamic scheduling can be performed based on the preset interruptibility attribute of the currently ongoing action trajectory. The first scheduling operation includes: if the interruptibility attribute of the action trajectory is interruptible, the next action trajectory can be played immediately. If the interruptibility attribute of the action trajectory is not interruptible, the action trajectory can continue to be executed, and the playback of subsequent action trajectories can be delayed. The delay time can be determined based on the time before the ongoing action trajectory has been played. If the delay time exceeds a set threshold, the execution of the action trajectory to be triggered can be canceled to ensure the smoothness and rationality of the overall response.

[0054] In some embodiments, step S19 includes: when the voice information is played to the corresponding trigger timestamp information, if there is an executing action trajectory, a corresponding second scheduling operation is performed based on the priority information of the executing action trajectory and the action trajectory in the target limb movement corresponding to the trigger timestamp information. Further, when the previous action has not finished playing and the next action is ready to be triggered, the priority information of the two action trajectories can be compared for dynamic scheduling. By comparing the priorities of the two, the action trajectory that needs to be played completely first is determined, and then the corresponding second scheduling operation is performed to ensure that the action trajectory with higher priority is played completely, while the action trajectory with lower priority can be interrupted or canceled. The priority information of the action trajectory can be determined based on preset quantification rules. For example, it can be determined based on the action attributes corresponding to the action trajectory (e.g., assigning different priorities to different attribute actions such as instruction-type (e.g., "nodding", "pointing at something") or auxiliary expression-type (e.g., hand waving while speaking)) or the priority of the action trajectory in the semantics of the response text. This embodiment does not limit the method of determining this priority information.

[0055] Figure 3 This diagram illustrates a device structure for generating robot limb movements according to an embodiment of this application. The device 1 includes a first module 11, a second module 12, a third module 13, and a fourth module 14. The first module 11 generates a response text and one or more action scripts corresponding to the interaction information using a large language model based on interaction information and prompt word information. The prompt word information includes several interaction examples, and the action scripts include action tag information. The second module 12 determines the action trajectory corresponding to each action script from an action library. The third module 13 uses a text-to-speech algorithm to determine the audio information corresponding to the response text. The audio information includes the speech information corresponding to the response text and the timestamp information corresponding to each character in the response text. The fourth module 14 determines a target limb movement matching the speech information based on the action trajectory corresponding to each action script and the audio information. The target limb movement includes one or more action trajectories and trigger timestamp information corresponding to each action trajectory. Figure 3 The specific implementation methods corresponding to modules 11, 12, 13 and 14 shown are the same as or similar to the specific embodiments of steps S11, S12, S13 and S14 mentioned above, and will not be repeated here. They are included here by reference.

[0056] In some embodiments, the device 1 further includes a five-module 15 (not shown). The five-module 15 determines the interaction information based on a voice activity detection model and an automatic speech recognition model. Here, the specific implementation of the five-module 15 is the same as or similar to the specific embodiment of the aforementioned step S15, and therefore will not be described again, but is incorporated herein by reference.

[0057] In some embodiments, the device 1 further includes a six-module 16 (not shown). The six-module 16 generates corresponding sample correction parameters based on the original action trajectories and sample sentiment information in the training dataset, using the multilayer perceptron model to be trained; determines corresponding corrected action trajectories based on the sample correction parameters and the original action trajectories; and updates the model parameters based on the corrected action trajectories and the sample action trajectories in the training dataset corresponding to the original action trajectories and sample sentiment information, combined with a corresponding loss function, to obtain the trained multilayer perceptron model. Here, the specific implementation of the six-module 16 is the same as or similar to the specific embodiment of the aforementioned step S16, and therefore will not be repeated, but is included herein by reference.

[0058] In some embodiments, the device 1 further includes a seven-module 17 (not shown). The seven-module 17 generates corresponding sample action trajectories based on the sample emotional information and the original action trajectory, using preset emotional style rules, and establishes a corresponding training dataset. Here, the specific implementation of the seven-module 17 is the same as or similar to the specific embodiment of the aforementioned step S17, and therefore will not be repeated here, but is incorporated herein by reference.

[0059] In some embodiments, the action script further includes action duration word count information, and the device 1 further includes an 8-module 18 (not shown). For each action script corresponding to an action trajectory, if the playback duration of the action trajectory is greater than the duration corresponding to the action duration word count information, the 8-module 18 updates the action trajectory. Here, the specific implementation of the 8-module 18 is the same as or similar to the specific embodiment of the aforementioned step S18, and therefore will not be repeated here, but is incorporated herein by reference.

[0060] In some embodiments, the device 1 further includes a one-nine module 19 (not shown). When the voice information is played to the corresponding trigger timestamp information, the one-nine module 19 executes the action trajectory of the target limb movement corresponding to the trigger timestamp information. Here, the specific implementation of the one-nine module 19 is the same as or similar to the specific embodiment of the aforementioned step S19, and therefore will not be described again, but is included herein by reference.

[0061] Figure 4 Exemplary systems that can be used to implement the various embodiments described in this application are shown; such as Figure 4As shown in some embodiments, system 300 can function as any of the devices described in each of the embodiments. In some embodiments, system 300 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 320) and one or more processors (e.g., one or more processors 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this application.

[0062] In one embodiment, the system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of the processors 305 and / or any suitable device or component communicating with the system control module 310.

[0063] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0064] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. In one embodiment, system memory 315 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 315 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0065] In one embodiment, the system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device 320 and (one or more) communication interfaces 325.

[0066] For example, NVM / storage device 320 may be used to store data and / or instructions. NVM / storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drive (HDD), one or more optical disc (CD) drives, and / or one or more digital universal optical disc (DVD) drives).

[0067] NVM / storage device 320 may include storage resources that are physically part of a device on which system 300 is mounted, or that can be accessed by the device without necessarily being part of it. For example, NVM / storage device 320 may be accessed via a network through one or more communication interfaces 325.

[0068] One or more communication interfaces 325 may provide the system 300 with an interface to communicate over one or more networks and / or with any other suitable device. The system 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0069] In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 (e.g., memory controller module 330). In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 to form a system-in-package (SiP). In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die. In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die to form a system-on-a-chip (SoC).

[0070] In various embodiments, system 300 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 300 may have more or fewer components and / or different architectures. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0071] In addition to the methods and devices described in the above embodiments, this application also provides a computer-readable storage medium storing computer code that, when executed, performs the method described in any of the preceding embodiments.

[0072] This application also provides a computer program product that, when executed by a computer device, performs the method described in any of the preceding claims.

[0073] This application also provides a computer device, the computer device comprising:

[0074] One or more processors;

[0075] Memory, used to store one or more computer programs;

[0076] When the one or more computer programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in any of the preceding methods.

[0077] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0078] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0079] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0080] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or hereafter developed capable of storing computer-readable information / data for use by a computer system.

[0081] Herein, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run a method and / or technical solution based on the foregoing embodiments of this application.

[0082] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0083] All data collected and processed in this application have been with the user's consent or permission, and have strictly complied with legal regulations, social ethics, and the public interest. This data processing includes, but is not limited to, tag management, rule setting, and recommendation decisions. The legal regulations mentioned include, but are not limited to: 1) relevant laws and regulations of various countries or organizations regarding the protection of personal information; 2) relevant laws and regulations of various countries or organizations regarding the protection of user information; and 3) relevant laws and regulations of various countries or organizations regarding the use of personal or user information by organizations.

Claims

1. A method for generating robot limb movements, characterized in that, The method includes: Based on the interaction information and prompt word information, a large language model is used to generate the response text corresponding to the interaction information and one or more action scripts. The prompt word information includes several interaction examples, and the action scripts include action tag information. Determine the action trajectory corresponding to each action script from the action library; Using a text-to-speech algorithm, the audio information corresponding to the reply text is determined, wherein the audio information includes the speech information corresponding to the reply text and the timestamp information corresponding to each character in the reply text; Based on the motion trajectory corresponding to each motion script and the audio information, a target limb action matching the voice information is determined, wherein the target limb action includes one or more motion trajectories and trigger timestamp information corresponding to each motion trajectory.

2. The method according to claim 1, characterized in that, The method further includes: The interaction information is determined based on the speech activity detection model and the automatic speech recognition model.

3. The method according to claim 1 or 2, characterized in that, The action script also includes emotion tagging information; Determining the action trajectory corresponding to each action script from the action library includes: For each action script, the corresponding action template is determined by querying the action library based on the action tag information; Based on the emotion tag information and the action template, the corresponding correction parameters are generated using a trained multilayer perceptron model; Based on the correction parameters and the action template, the action trajectory corresponding to the action script is determined.

4. The method according to claim 3, characterized in that, The method further includes: Based on the original motion trajectories and sample sentiment information in the training dataset, corresponding sample correction parameters are generated using the multilayer perceptron model to be trained. Based on the sample correction parameters and the original action trajectory, the corresponding corrected action trajectory is determined; Based on the corrected action trajectory and the sample action trajectories in the training dataset that correspond to the original action trajectory and sample sentiment information, the model parameters are updated by combining the corresponding loss function to obtain the trained multilayer perceptron model.

5. The method according to claim 4, characterized in that, The method further includes: Based on the sample sentiment information and the original action trajectory, corresponding sample action trajectories are generated using preset sentiment style rules, and a corresponding training dataset is established.

6. The method according to any one of claims 1 to 5, wherein, The action script also includes information on the number of characters for the action duration; The method further includes: For each action script corresponding to an action trajectory, if the playback duration of the action trajectory is greater than the duration corresponding to the number of characters in the action duration information, the action trajectory is updated.

7. The method according to any one of claims 1 to 6, wherein, The method further includes: When the voice message is played to the corresponding trigger timestamp information, the action trajectory of the target limb action corresponding to the trigger timestamp information is executed.

8. The method according to claim 7, wherein, The step of executing the action trajectory corresponding to the trigger timestamp information in the target limb movement when the voice information is played includes: When the voice message is played to the corresponding trigger timestamp information, if there is an action trajectory being executed, the execution of that action trajectory is terminated, and the action trajectory corresponding to the trigger timestamp information in the target limb action is executed.

9. The method according to claim 7, wherein, The step of executing the action trajectory corresponding to the trigger timestamp information in the target limb movement when the voice information is played includes: When the voice information is played to the corresponding trigger timestamp information, if there is an ongoing action trajectory, the corresponding first scheduling operation is executed based on the interruptible attribute of the ongoing action trajectory.

10. The method according to claim 7, wherein, The step of executing the action trajectory corresponding to the trigger timestamp information in the target limb movement when the voice information is played includes: When the voice information is played to the corresponding trigger timestamp information, if there is an ongoing action trajectory, a corresponding second scheduling operation is performed based on the priority information of the ongoing action trajectory and the action trajectory in the target limb movement corresponding to the trigger timestamp information.

11. A computer device for generating robot limb movements, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 10.

Images