Robot control model training method, control method, device and electronic equipment

Abstract

The application provides a robot control model training method, a control method, a device and electronic equipment, and relates to the technical field of device control. The method comprises the following steps: obtaining a preset action data set, using a preset large language model in a robot control model, generating an action sequence text corresponding to a sample natural language command and a predicted target direction based on the sample natural language command; using an initial text encoder in the robot control model to encode the action sequence text, determining a target action code corresponding to the action code from a preset action codebook, using an initial action planning model in the robot control model to plan an action according to the target action code and the predicted target direction, and generating action strategy information corresponding to the sample natural language command; and performing parameter training on the initial action planning model in the robot control model. The application improves the generalization ability of the robot control model, and makes the control of the robot more accurate.

Description

Technical Field

[0001] This application relates to the field of equipment control technology, and more specifically, to a training method, control method, device, and electronic equipment for a robot control model. Background Technology

[0002] The development of humanoid robot control technology relies on the deep integration of reinforcement learning and imitation learning, aiming to achieve autonomous execution of complex tasks. In recent years, generative adversarial imitation learning has provided an effective framework for policy training by distinguishing expert trajectories from policy-generated trajectories through discriminators. However, it can only generate actions for specific tasks and is difficult to cope with dynamic and unknown scenarios.

[0003] In traditional methods, generative adversarial imitation learning trains a policy and discriminator to mimic expert behavior and generate actions. Alternatively, adversarial skill embedding utilizes a hierarchical model to enable low-level policies to learn reusable skills, which are then trained to higher-level policies to solve tasks. Adversarial motion priors learn to imitate different skills from reference data to complete higher-level tasks.

[0004] However, large language models suffer from the contradiction between the randomness of model output and the precision of instruction execution, lack effective text quantization methods, task specificity caused by multi-strategy dependence, inability to handle unknown tasks with a single strategy, and lack of a general mechanism that takes into account both action imitation and direction control in motion control. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a training method, control method, device, and electronic equipment for robot control models, thereby improving the generalization ability of robot control models and making robot control more precise.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, one embodiment of this application provides a method for training a robot control model, the method comprising:

[0008] Obtain a preset action dataset, which includes: sample natural language commands and preset reference actions corresponding to the sample natural language commands;

[0009] Using a pre-defined large language model in the robot control model, the action sequence text and predicted target direction corresponding to the sample natural language command are generated based on the sample natural language command.

[0010] The action sequence text is encoded using the initial text encoder in the robot control model to obtain the action code corresponding to the sample natural language command;

[0011] Based on the action code, the target action code corresponding to the action code is determined from a preset action codebook, wherein the preset action codebook stores multiple preset action codes in advance;

[0012] Using the initial motion planning model in the robot control model, motion planning is performed based on the target motion code and the predicted target direction to generate motion strategy information corresponding to the sample natural language command;

[0013] Based on the preset reference action and the target robot's action state information under the action strategy information, the initial action planning model in the robot control model is trained by parameter tuning.

[0014] Optionally, the initial action planning model includes: a skill encoder, an initial policy network, and a discriminator;

[0015] The initial motion planning model in the robot control model is used to perform motion planning based on the target motion code and the predicted target direction, generating motion strategy information corresponding to the sample natural language command, including:

[0016] The skill encoder is used to perform skill encoding on the target action code to obtain the action vector corresponding to the sample natural language command;

[0017] Using the initial policy network, action policy information corresponding to the sample natural language command is generated based on the predicted target direction and the action vector;

[0018] The step of parameter tuning and training the initial motion planning model in the robot control model based on the preset reference motion and the motion state information of the target robot under the motion strategy information includes:

[0019] Using the discriminator, the action reward parameters under the sample natural language command are calculated based on the preset reference action and the action state information of the target robot under the action strategy information.

[0020] The initial policy network is trained and its parameters are adjusted based on the action reward parameters.

[0021] Optionally, the action dataset further includes: a preset target direction corresponding to the sample natural language command; before tuning the initial policy network according to the action reward parameters, the method further includes:

[0022] Obtain the actual movement direction of the target robot under the action strategy information;

[0023] Calculate the task reward parameters under the sample natural language command based on the preset target direction and the actual action direction;

[0024] The step of tuning and training the initial policy network based on the action reward parameters includes:

[0025] The initial policy network is trained by adjusting the parameters based on the action reward parameters and the task reward parameters.

[0026] Optionally, the method further includes:

[0027] Calculate the encoding loss parameter based on the action code corresponding to the sample natural language command and the corresponding target action code;

[0028] The initial text encoder is trained by adjusting the parameters based on the encoding loss parameters.

[0029] Optionally, determining the target action code corresponding to the action code from a preset action codebook based on the action code includes:

[0030] Based on the action code, calculate the similarity between it and each preset action code in the preset action codebook;

[0031] Based on the similarity, the preset action code that is closest to the action code is determined from the preset action codebook as the target action code.

[0032] Optionally, the step of using a preset large language model in the robot control model to generate the action sequence text and predicted target direction corresponding to the sample natural language command based on the sample natural language command includes:

[0033] Based on the sample natural language commands, the description information of the target robot, the description information of the scene in which the target robot is located, and the basic skill information of the target robot, prompt words are generated;

[0034] Using the preset large language model, the action sequence text and the predicted target direction are generated based on the prompt words.

[0035] Secondly, another embodiment of this application provides a robot control method, the method comprising:

[0036] Obtain input natural language commands;

[0037] Using a pre-defined large language model in the robot control model, the action sequence text and predicted target direction corresponding to the natural language command are generated based on the natural language command.

[0038] The action sequence text is encoded using the initial text encoder in the robot control model to obtain the action code corresponding to the natural language command;

[0039] Based on the action code, the target action code corresponding to the action code is determined from a preset action codebook, wherein the preset action codebook stores multiple preset action codes in advance;

[0040] Using the target action planning model in the robot control model, action planning is performed based on the target action encoding and the predicted target direction to generate action strategy information corresponding to the sample natural language command; wherein, the robot control model is a model trained using any of the robot control model training methods described in the first aspect above;

[0041] The robot is controlled based on the predicted action strategy information.

[0042] Secondly, another embodiment of this application provides a training device for a robot control model, the device comprising:

[0043] The acquisition module is used to acquire a preset action dataset, which includes: sample natural language commands and preset reference actions corresponding to the sample natural language commands.

[0044] The generation module is used to generate the action sequence text and predicted target direction corresponding to the sample natural language command based on the sample natural language command by using a preset large language model in the robot control model.

[0045] The encoding module is used to encode the action sequence text using the initial text encoder in the robot control model to obtain the action code corresponding to the sample natural language command;

[0046] The determining module is used to determine the target action code corresponding to the action code from a preset action codebook based on the action code, wherein the preset action codebook stores multiple preset action codes in advance;

[0047] The planning module is used to use the initial motion planning model in the robot control model to perform motion planning based on the target motion code and the predicted target direction, and generate motion strategy information corresponding to the sample natural language command;

[0048] The training module is used to perform parameter tuning training on the initial motion planning model in the robot control model based on the preset reference motion and the motion state information of the target robot under the motion strategy information.

[0049] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the methods described in either the first or second aspect above.

[0050] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the methods described in either the first or second aspect above.

[0051] The beneficial effects of this application are:

[0052] This application provides a training method, control method, device, and electronic device for a robot control model. It involves acquiring a preset action dataset, employing a preset large language model within the robot control model to generate action sequence text and predicted target directions based on sample natural language commands; encoding the action sequence text using an initial text encoder within the robot control model to obtain action codes corresponding to the sample natural language commands; determining the target action code corresponding to the action code from a preset action codebook, which pre-stores multiple preset action codes; using an initial action planning model within the robot control model to perform action planning based on the target action code and predicted target direction, generating action strategy information corresponding to the sample natural language commands; and training the initial action planning model within the robot control model using preset reference actions and the target robot's action state information under the action strategy information. This application generates corresponding action sequence text and predicted target directions using a preset large language model based on the preset action dataset. By understanding the corresponding tasks through the preset large language model, it can adapt to different scenarios and improve the understanding of natural language. The initial text encoder in the robot control model is used to encode the action sequence text to obtain the target action code, which standardizes the action sequence text and adapts it to diverse tasks. The initial action planning model in the robot control model generates action strategy information, ensuring accurate generation of this information and dynamic adaptation to the environment and robot state. Based on preset reference actions and the target robot's action state information under the action strategy information, parameter tuning training of the initial action planning model in the robot control model can improve the generalization ability of the robot control model. The method of this application can ensure the accuracy of generated action strategy information, thereby improving the generalization ability of the robot control model and making robot control more precise. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for training a robot control model provided in an embodiment of this application;

[0055] Figure 2 A schematic diagram illustrating the initial policy network training process in a robot model training method provided in this application embodiment;

[0056] Figure 3 A schematic diagram illustrating the initial policy network training process in another robot model training method provided in this application embodiment;

[0057] Figure 4 This is a flowchart illustrating the parameter tuning of the initial text encoder in a robot model training method provided in this application embodiment.

[0058] Figure 5 This is a schematic diagram illustrating the process of determining the target action encoding in a training method for a robot control model provided in an embodiment of this application.

[0059] Figure 6 The process of determining text and direction in a robot control model training method provided in this application embodiment;

[0060] Figure 7 This is a schematic diagram of the structure of a robot control model provided in an embodiment of this application;

[0061] Figure 8 A scene diagram illustrating the robot control method provided in the embodiments of this application;

[0062] Figure 9 A flowchart illustrating a robot control method based on a robot control model, provided as an embodiment of this application;

[0063] Figure 10 A schematic diagram of the structure of a training device for a robot control model provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0066] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0067] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0068] During robot operation, the working environment is often complex and changeable, with many uncertainties. For example, in dynamic scenes, robots may encounter different terrains, interference, and real-time adjustments to task requirements. Therefore, robot control needs to be highly adaptable, able to quickly perceive environmental changes and make corresponding adjustments. It also needs efficient control strategies to execute tasks accurately and smoothly under complex conditions. Currently, reinforcement learning and imitation learning are used to control robots. However, existing methods require creating simulated environments and rewards for specific tasks, resulting in the need for multiple strategies and an inability to handle complex and unknown tasks. Therefore, this application provides a training method for a robot control model. The method involves acquiring a pre-set action dataset, using a pre-set large language model within the robot control model to generate action sequence text and predicted target direction corresponding to sample natural language commands; encoding the action sequence text using an initial text encoder within the robot control model to obtain action codes corresponding to the sample natural language commands; determining the target action code corresponding to the action code from a pre-set action codebook, wherein the pre-set action codebook stores multiple pre-set action codes; using an initial action planning model within the robot control model to perform action planning based on the target action code and predicted target direction, generating action strategy information corresponding to the sample natural language commands; and training the initial action planning model within the robot control model by adjusting parameters based on pre-set reference actions and the target robot's action state information under the action strategy information. The robot control model trained using this method can control a robot, controlling the robot based on the predicted action strategy information obtained from the robot control model. The robot control model in this application is a general model that can control the robot based solely on predicted action strategy information, without the need to create a corresponding simulation environment for a specific task, thereby improving the efficiency of robot control and reducing the complexity of robot control.

[0069] The following description, in conjunction with the accompanying drawings, illustrates the training method for the robot control model provided in this application. This method can be executed by an electronic device, which includes a memory and a processor. The processor can be a local controller of the robot or a cloud server. The processor is used to execute the robot motion control model training method and the robot control method. Figure 1 This is a flowchart illustrating a method for training a robot control model according to an embodiment of this application. Figure 1 As shown, the method includes:

[0070] Step 101: Obtain the preset action dataset.

[0071] The action dataset includes: sample natural language commands and corresponding preset reference actions. The sample natural language commands are the control instructions for the robot, such as "Please walk to the kitchen door". The preset reference actions corresponding to the sample natural language commands can be reference actions obtained from a professional database. Specifically, they can include action sequences of specific actions. For example, when the feature action is to lift a leg, the preset reference action corresponds to a series of actions of lifting the leg.

[0072] Optionally, the preset action dataset can be data obtained from a preset model training dataset, or it can be a dataset created by the robot control model itself according to its needs. This application does not impose any restrictions on this.

[0073] Step 102: Using the preset large language model in the robot control model, generate the action sequence text and predicted target direction corresponding to the sample natural language commands based on the sample natural language commands.

[0074] The action sequence text includes multiple actions and their corresponding predicted target directions. For example, the action sequence may include: turn right, run forward, left, walk forward, turn right, and walk forward. The predicted target direction can be the corresponding three-dimensional coordinates, such as [0, 1, 2]. The default large language model is the Large Language Model (LLM).

[0075] Optionally, a pre-set large language model in the robot control model is used to generate the action sequence text and predicted target direction corresponding to the sample natural language command based on the pre-set prompt words and the current scene context information.

[0076] For example, when the sample natural language command is "Please walk to the kitchen door", the preset large language model in the robot control model generates the action sequence text and predicted target direction of "turn right [0, 1, 2], run forward [0, 1, 2], turn left [0, 1, 5], walk forward [0, 1, 5], turn right [0, 6, 5] and walk forward [0, 6, 5]" based on "Please walk to the kitchen door".

[0077] Step 103: Use the initial text encoder in the robot control model to encode the action sequence text to obtain the action code corresponding to the sample natural language command.

[0078] The initial text encoder can be a Contrastive Language-Image Pretraining (CLIP) text encoder. Specifically, the initial text encoder converts the action sequence text into a 512-dimensional vector.

[0079] Optionally, an initial text encoder is used to encode each action in the action sequence text to obtain a 512-dimensional vector for each action. The set of 512-dimensional vectors for each action is the action code corresponding to the sample natural language command.

[0080] Step 104: Based on the action code, determine the target action code corresponding to the action code from the preset action codebook.

[0081] The preset action codebook stores multiple preset action codes. The preset action codebook can be a 512-dimensional vector corresponding to the text labels in the motion dataset used during training. The motion dataset contains action-related data from the robot's training process, with each action having a corresponding text label.

[0082] Optionally, based on the action code corresponding to each action, the Euclidean distance between each action code and multiple preset action codes in the preset action codebook is calculated, and the preset action code that is closest to the action code in the preset action codebook is used as the target action code for that action.

[0083] For example, the action in the action sequence text is "turn right". The initial text encoder is used to obtain the corresponding 512-dimensional vector. The Euclidean distance between the 512-dimensional vector corresponding to "turn right" and multiple preset action codes in the preset action codebook is calculated. The preset action code with the closest distance is "turn right" in the motion dataset, which is obtained by using the initial text encoder to obtain the corresponding 512-dimensional vector.

[0084] Step 105: Using the initial motion planning model in the robot control model, motion planning is performed based on the target motion code and the predicted target direction to generate motion strategy information corresponding to the sample natural language commands.

[0085] The initial motion planning model is used to generate motion strategy information based on the target motion encoding and the predicted target direction. The motion strategy information is used to control the movement of the target robot.

[0086] Step 106: Based on the preset reference action and the target robot's action state information under the action strategy information, perform parameter tuning and training on the initial action planning model in the robot control model.

[0087] Optionally, based on the preset reference action and the target robot's action state information under the action policy information, the difference between the target robot's action state information under the action policy information and the preset reference action is determined, and the initial action planning model in the robot control model is trained by adversarial imitation learning.

[0088] In this embodiment, a preset action dataset is obtained. A preset large language model in the robot control model is used to generate the action sequence text and predicted target direction corresponding to the sample natural language commands. An initial text encoder in the robot control model is used to encode the action sequence text to obtain the action codes corresponding to the sample natural language commands. Based on the action codes, the target action codes corresponding to the action codes are determined from a preset action codebook, which pre-stores multiple preset action codes. An initial action planning model in the robot control model is used to perform action planning based on the target action codes and predicted target directions, generating action strategy information corresponding to the sample natural language commands. Based on preset reference actions and the target robot's action state information under the action strategy information, the initial action planning model in the robot control model is trained and its parameters are adjusted. In this application, a preset large language model is used to generate the corresponding action sequence text and predicted target direction based on the preset action dataset. Understanding the corresponding task through the preset large language model allows for adaptation to different scenarios and improves the understanding of natural language. Encoding the action sequence text with the initial text encoder in the robot control model to obtain the target action codes standardizes the action sequence text, thereby adapting it to diverse tasks. This paper employs an initial motion planning model within the robot control model to generate motion policy information. This ensures the accurate generation of the motion policy information and allows for dynamic adaptation to the environment and robot state. By tuning the initial motion planning model based on preset reference actions and the target robot's motion state information under the motion policy information, the generalization ability of the robot control model can be improved. The method in this application guarantees the accuracy of the generated motion policy information, thereby enhancing the generalization ability of the robot control model and enabling more precise robot control.

[0089] Based on the above embodiments, the initial motion planning model includes: a skill encoder, an initial policy network, and a discriminator. This application also provides a process for training the initial policy network in a robot model training method. Figure 2 This is a schematic diagram illustrating the initial policy network training process in a robot model training method provided in an embodiment of this application, as shown below. Figure 2 As shown, in step 105 above, the initial motion planning model in the robot control model is used to perform motion planning based on the target motion encoding and predicted target direction, generating motion strategy information corresponding to the sample natural language commands, including:

[0090] Step 201: Use a skill encoder to perform skill encoding on the target action code to obtain the action vector corresponding to the sample natural language command.

[0091] The skill encoder is used to reduce the dimensionality of high-dimensional text features in target action encoding to low-dimensional action vectors. The action vectors are used to represent action semantics.

[0092] Step 202: Using the initial policy network, generate action policy information corresponding to the sample natural language command based on the predicted target direction and action vector.

[0093] Optionally, an initial policy network is used to transform the predicted target direction and action vector into action policy information that the robot can execute, based on the predicted target direction, action vector, and current scene context information. The current scene context information may include the motion status of each joint of the robot and environmental information. The motion status of each joint may include information such as the robot's height, the position of each limb, the speed of each limb, and the current orientation. Environmental information may include whether the ground is flat and whether there are obstacles in the current environment. This application embodiment does not limit the specific content of the current scene context information.

[0094] In step 106 above, based on the preset reference action and the target robot's action state information under the action policy information, the initial action planning model in the robot control model is trained by parameter tuning, including:

[0095] Step 203: Using a discriminator, calculate the action reward parameters under the sample natural language command based on the preset reference action and the action state information of the target robot under the action strategy information.

[0096] The preset reference motion includes the robot's preset joint states and preset motion trajectories; the target robot's motion state information under the motion strategy information includes the robot's joint states and motion trajectories under the motion strategy information.

[0097] Optionally, based on the state of the target robot not executing the action strategy information. st and the state after executing the action strategy information. st +1 Determines the actual movement direction of the target robot, and determines the state of unexecuted action strategies in the preset action dataset based on the preset action dataset. st and the state after executing the action strategy information. st +1 corresponds to the preset target direction. The state transition distribution between the preset target direction and the actual action direction is distinguished by the discriminator target function.

[0098] Optionally, the objective function is:

[0099]

[0100] in, For discriminator, The discriminator parameters used to find the parameters that minimize the objective function are E[•], which is used to calculate the mathematical expectation. The state transition distribution corresponding to the preset target direction, Let Z represent the state transition distribution corresponding to the actual action direction, log(•) be the logarithmic function, and Z be the action strategy information corresponding to the sample natural language command. For gradient penalty weights, Input variables for the discriminator , For the discriminator to input variables The gradient vector.

[0101] Optionally, the action reward parameters of the discriminator are determined based on the objective function. If the difference between the preset target direction and the actual movement direction is greater, then The closer it is to 0, the smaller the difference between the preset target direction and the actual action direction. The closer it is to 1.

[0102] Step 204: Train the initial policy network by adjusting its parameters based on the action reward parameters.

[0103] Optionally, the feedback signal of the discriminator is generated based on the action reward parameters, and the initial policy network is trained by parameter tuning, so that the action state information corresponding to the action policy information obtained by the parameter-tuned policy network is closer to the preset reference action.

[0104] In this embodiment, a skill encoder is used to obtain the action vector corresponding to the sample natural language command based on the target action encoding; an initial policy network is used to generate action policy information corresponding to the sample natural language command; a discriminator is used to calculate the action reward parameters under the sample natural language command; and the initial policy network is trained and tuned based on the action reward parameters. This application can efficiently train and optimize resources, improving the accuracy of the action policy information generated by the policy network.

[0105] Based on the above embodiments, the action dataset further includes: a preset target direction corresponding to the sample natural language commands. This application also provides a process for training the initial policy network in another robot model training method. Figure 3 A schematic diagram illustrating the initial policy network training process in another robot model training method provided in this application embodiment is shown below. Figure 3 As shown, before performing parameter tuning and training on the initial policy network in step 204 above, the method further includes:

[0106] Step 301: Obtain the actual movement direction of the target robot under the action strategy information.

[0107] The actual movement direction of the target robot under the action strategy information includes the direction vectors of the robot's pelvis, left hip bone, and right hip bone. When the target robot is holding other objects, the actual movement direction of the target robot under the action strategy information also includes parameters such as the position, velocity, rotation relative to the pelvis, and angular velocity of the other objects. The other objects can be any objects, such as handkerchiefs or shields, and this application embodiment does not impose any restrictions on this.

[0108] Optionally, inertial sensors are set on the pelvis, left hip joint, and right hip joint of the target robot to obtain the actual movement direction of the target robot under the action strategy information.

[0109] Step 302: Calculate the task reward parameters under the sample natural language command based on the preset target direction and the actual action direction.

[0110] The preset target direction includes the preset direction vectors of the robot's pelvis, left hip bone, and right hip bone. When the target robot is holding other objects, the actual movement direction of the target robot under the action strategy information also includes parameters such as the position, velocity, rotation relative to the pelvis, and angular velocity of the other objects. The other objects can be any objects, such as a handkerchief or a shield, and this application embodiment does not impose any restrictions on this.

[0111] Optionally, a pelvic reward function is calculated based on the actual robot's pelvic orientation vector, a preset robot pelvic orientation vector, and a preset pelvic weight coefficient. A left hip bone reward function is calculated based on the actual robot's left hip bone orientation vector, a preset robot left hip bone orientation vector, and a preset left hip bone weight coefficient. A right hip bone reward function is calculated based on the actual robot's right hip bone orientation vector, a preset robot right hip bone orientation vector, and a preset right hip bone weight coefficient. The sum of the preset pelvic weight coefficient, left hip bone weight coefficient, and right hip bone weight coefficient is 1.

[0112] Optionally, the task reward parameters are determined based on the pelvic reward function, the left hip bone reward function, and the right hip bone reward function. Specifically, the sum of the pelvic reward function, the left hip bone reward function, and the right hip bone reward function is used as the task reward parameter.

[0113] In step 204 above, the initial policy network is trained by adjusting the parameters based on the action reward parameters, including:

[0114] Step 303: Train the initial policy network by adjusting the parameters based on the action reward parameters and the task reward parameters.

[0115] Optionally, the reward function of the initial policy network is determined based on the action reward parameters and their corresponding weights, and the task reward parameters and their corresponding weights. The initial policy network is then trained using this reward function. The sum of the action reward parameter weights and the task reward parameter weights is 1.

[0116] In this embodiment, the initial policy network is trained by adjusting the parameters based on action reward parameters and task reward parameters to obtain the policy network. This application integrates imitation and physical orientation control as optimization objectives, which can both preserve the action accuracy of imitation learning and provide an efficient and robust action strategy for humanoid robot control in complex scenarios through the physical guidance of task rewards.

[0117] Based on the above embodiments, this application also provides a process for tuning the initial text encoder in a robot control model training method. Figure 4 This is a flowchart illustrating the parameter tuning of the initial text encoder in a robot model training method provided in this application embodiment, as shown below. Figure 4 As shown, based on steps 101-106 above, the method further includes:

[0118] Step 401: Calculate the encoding loss parameters based on the action codes corresponding to the sample natural language commands and the corresponding target action codes.

[0119] Optionally, the Euclidean distance in 512-dimensional space is calculated based on the action code corresponding to the sample natural language command and the corresponding target action code, thereby obtaining the coding loss parameter.

[0120] For example, based on the sample natural language command "quickly kick forward", the output is a 512-dimensional vector of the action code corresponding to the sample natural language command, and the encoder's target action code is a 512-dimensional vector corresponding to "kick". Thus, the encoding loss parameters are obtained based on the action code corresponding to the sample natural language command and the corresponding target action code.

[0121] For example, given the sample natural language command "quickly kick forward", the output is a 512-dimensional vector of the action encoding corresponding to the sample natural language command. The label corresponding to the sample natural language command is encoded into a 512-dimensional vector by the encoder. Based on the 512-dimensional vector of the action encoding and the 512-dimensional vector of the label, the encoding loss function is calculated. The label corresponding to the sample natural language command can be a pre-stored label of the sample natural language command in a preset action dataset.

[0122] Step 402: Based on the encoding loss parameters, perform parameter tuning training on the initial text encoder.

[0123] Optionally, the parameters of the initial text encoder are adjusted through backpropagation according to the encoding loss function, thereby reducing the loss function of the initial text encoder.

[0124] In this embodiment, encoding loss parameters are calculated based on the action codes corresponding to sample natural language commands and the corresponding target action codes, and the initial text encoder is trained using these parameters. This application can significantly improve the text encoder's understanding of natural language, thereby increasing the accuracy of the action codes corresponding to the obtained sample natural language commands.

[0125] Based on the above embodiments, this application also provides a process for determining the target action encoding in a robot control model training method. Figure 5 This is a schematic diagram illustrating the process of determining the target action encoding in a robot control model training method provided in an embodiment of this application, as shown below. Figure 5 As shown, in step 104 above, the target action code corresponding to the action code is determined from the preset action codebook based on the action code, including:

[0126] Step 501: Calculate the similarity between the action code and each preset action code in the preset action codebook.

[0127] Optionally, the geometric distance between the action code and each preset action code in the preset action codebook in the multidimensional space can be calculated to obtain the similarity between the action code and each preset action code in the preset action codebook.

[0128] Step 502: Based on similarity, determine the preset action code that is closest to the action code from the preset action codebook as the target action code.

[0129] Optionally, the preset action code with the highest similarity to the action code is determined from the preset action codebook as the target action code. The highest similarity is defined as the preset action code with the smallest geometric distance among all preset action codes in the preset action codebook.

[0130] In this embodiment, the similarity between the action code and each preset action code in the preset action codebook is calculated. Based on the similarity, the preset action code that is closest to the action code in the preset action codebook is determined as the target action code. In this embodiment, determining the target action code based on similarity can reduce computational complexity and improve training and inference efficiency.

[0131] Based on the above embodiments, this application also provides a process for determining text and direction in a robot control model training method. Figure 6 The process for determining text and direction in a robot control model training method provided in this application embodiment is as follows: Figure 6 As shown, in step 101 above, a preset large language model in the robot control model is used to generate the action sequence text and predicted target direction corresponding to the sample natural language commands based on the sample natural language commands, including:

[0132] Step 601: Generate prompt words based on the sample natural language commands, the description information of the target robot, the description information of the scene in which the target robot is located, and the basic skill information of the target robot.

[0133] The description information of the target robot can be the basic physical attributes of the target robot, such as: body structure and shape, as well as equipment and tools. The description information of the target robot can be "humanoid robot with two arms and two legs, a simple gripper on the right hand, and no equipment on the left hand".

[0134] The scene description information for the target robot can include environmental details, spatial structure, and other information. For example, the scene description information for the target robot could be "Kitchen scene: Dishes and bowls are scattered on the table".

[0135] The basic skill information of the target robot can include motor skill limitations, such as the boundaries of its motor capabilities and the basic skills it has already mastered. The basic skill information of the target robot can be "able to walk forward and backward, turn left and right, and grasp small objects with its right gripper".

[0136] The prompt can be an action and the coordinates corresponding to the action.

[0137] Optionally, the target robot's description information, the scene description information where the target robot is located, and the target robot's basic skill information are used to determine the actions that the target robot can perform, and corresponding prompt words are generated based on the sample natural language commands.

[0138] Step 602: Using a pre-set large language model, generate action sequence text and predict target direction based on prompt words.

[0139] In this embodiment, a pre-defined large language model is used to generate action sequence text and predict target direction based on prompt words. This application can guarantee the accuracy of the generated action sequence text and predicted target direction, and can also guarantee that the action sequence text and predicted target direction are actions that the target robot can perform, thereby improving the control efficiency of the target robot.

[0140] The training method of the robot control model provided in the embodiments of this application will be described below in conjunction with the robot control model. Figure 7 This is a schematic diagram of the structure of a robot control model provided in an embodiment of this application, as shown below. Figure 7As shown, the robot control model includes: a pre-set large language model, an initial text encoder, and an initial motion planning model. The initial motion planning model includes: a skill encoder, an initial policy network, and a discriminator. Specifically, the pre-set large language model in the robot control model generates the action sequence text and predicted target direction corresponding to the sample natural language command based on the sample natural language command. The initial text encoder and a pre-set action codebook in the robot control model encode the action sequence text to obtain the target action code corresponding to the sample natural language command. The skill encoder performs skill encoding on the target action code to obtain the action vector corresponding to the sample natural language command. The initial policy network generates action policy information corresponding to the sample natural language command based on the predicted target direction and action vector. The discriminator calculates the action reward parameters under the sample natural language command based on the pre-set reference action and the target robot's action state information under the action policy information. The initial policy network is then trained and tuned based on the action reward parameters.

[0141] Based on the above embodiments, this application provides a robot control method based on a robot control model. Figure 8 This is a scene diagram of the robot control method provided in the embodiments of this application. Figure 9 A flowchart illustrating a robot control method based on a robot control model, as provided in this application embodiment, is shown below. Figure 8 As shown, this application uses a domestic robot as an example. This robot is humanoid, with two arms and two legs, a simple gripper on its right hand, and no equipment on its left hand. The current scene is a kitchen scene, including a table with dishes scattered on it. The robot can walk forward and backward, turn left and right, and can use its right gripper to grasp small objects. Figure 9 As shown, the method includes:

[0142] Step 901: Obtain the input natural language command.

[0143] Natural language commands can be entered manually, obtained through voice recognition, or determined by selecting preset functions.

[0144] Step 902: Using the preset large language model in the robot control model, generate the action sequence text and predicted target direction corresponding to the natural language command based on the natural language command.

[0145] Optionally, a pre-set large language model in the robot control model is used to generate the action sequence text and predicted target direction corresponding to the sample natural language command based on the sample natural language command and the current scene context information.

[0146] Step 903: Encode the action sequence text using the initial text encoder in the robot control model to obtain the action code corresponding to the natural language command.

[0147] Optionally, an initial text encoder is used to encode each action in the action sequence text to obtain a 512-dimensional vector for each action. The set of 512-dimensional vectors for each action is the action code corresponding to the sample natural language command.

[0148] Step 904: Based on the action code, determine the target action code corresponding to the action code from the preset action codebook.

[0149] The preset action codebook contains multiple preset action codes.

[0150] Optionally, based on the action code corresponding to each action, the Euclidean distance between each action code and multiple preset action codes in the preset action codebook is calculated, and the preset action code with the closest distance to the action code is obtained as the target action code for that action.

[0151] Step 905: Using the target motion planning model in the robot control model, motion planning is performed based on the target motion encoding and predicted target direction to generate motion strategy information corresponding to the sample natural language commands.

[0152] The robot control model is a model trained using any robot control model training method.

[0153] Step 906: Control the robot according to the predicted action strategy information.

[0154] Optionally, the robot can be controlled using predicted action strategy information, so that the robot moves based on the predicted action strategy.

[0155] In this embodiment, the following steps are taken: First, an input natural language command is acquired. Then, a pre-set large language model within the robot control model is used to generate the corresponding action sequence text and predicted target direction based on the natural language command. Next, an initial text encoder within the robot control model is used to encode the action sequence text, obtaining the action code corresponding to the natural language command. Based on the action code, the target action code corresponding to the action code is determined from a pre-set action codebook. Finally, a target action planning model within the robot control model is used to perform action planning based on the target action code and predicted target direction, generating action strategy information corresponding to the sample natural language command. Finally, the robot is controlled based on the predicted action strategy information. This application ensures the efficiency of generating the action sequence text and predicting the target direction through the pre-set large language model. Determining the action strategy information through the target action planning model ensures that the robot executes the corresponding actions in sequence, guaranteeing the accuracy and efficiency of robot control.

[0156] Based on the same inventive concept, this application also provides a robot control model training device corresponding to the robot control model training method. Since the principle of the device in this application is similar to the robot control model training method described above, the implementation of the device can refer to the implementation of the method.

[0157] Figure 10 This is a schematic diagram of the structure of a training device for a robot control model provided in an embodiment of this application, as shown below. Figure 10 As shown, the device includes: an acquisition module 1001, a generation module 1002, an encoding module 1003, a determination module 1004, a planning module 1005, and a training module 1006; wherein, the acquisition module 1001 is used to acquire a preset action dataset, which includes: sample natural language commands and preset reference actions corresponding to the sample natural language commands.

[0158] The generation module 1002 is used to generate the action sequence text and predicted target direction corresponding to the sample natural language command based on the sample natural language command by using the preset large language model in the robot control model.

[0159] The encoding module 1003 is used to encode the action sequence text using the initial text encoder in the robot control model to obtain the action code corresponding to the sample natural language command;

[0160] The determining module 1004 is used to determine the target action code corresponding to the action code from the preset action code book according to the action code, wherein the preset action code book stores multiple preset action codes in advance;

[0161] Planning module 1005 is used to adopt the initial motion planning model in the robot control model, perform motion planning based on the target motion code and predicted target direction, and generate motion strategy information corresponding to sample natural language commands;

[0162] The training module 1006 is used to perform parameter tuning training on the initial motion planning model in the robot control model based on the preset reference motion and the motion state information of the target robot under the motion strategy information.

[0163] Optionally, the initial action planning model includes: a skill encoder, an initial policy network, and a discriminator; the planning module 1005 is specifically used to: use the skill encoder to perform skill encoding on the target action encoding to obtain the action vector corresponding to the sample natural language command;

[0164] An initial policy network is used to generate action policy information corresponding to sample natural language commands based on the predicted target direction and action vector.

[0165] Training module 1006 is specifically used to: use a discriminator to calculate the action reward parameters under the sample natural language command based on the preset reference action and the action state information of the target robot under the action policy information;

[0166] The initial policy network is trained by adjusting the parameters based on the action reward parameters.

[0167] Optionally, the action dataset also includes: a preset target direction corresponding to the sample natural language command; the training module 1006 is also used to: obtain the actual action direction of the target robot under the action policy information;

[0168] Calculate the task reward parameters under the sample natural language commands based on the preset target direction and the actual action direction;

[0169] Training module 1006 is specifically used for: tuning and training the initial policy network based on action reward parameters and task reward parameters.

[0170] Optionally, the device further includes a calculation module, which is specifically used to calculate the encoding loss parameter based on the action code corresponding to the sample natural language command and the corresponding target action code;

[0171] The initial text encoder is trained by adjusting the parameters based on the encoding loss parameters.

[0172] Optionally, the encoding module 1005 is specifically used to: calculate the similarity between the action code and each preset action code in the preset action codebook, based on the action code;

[0173] Based on similarity, the preset action code that is closest to the action code is determined from the preset action codebook as the target action code.

[0174] Optionally, the generation module 1002 is specifically used to: generate prompt words based on the sample natural language commands, the description information of the target robot, the description information of the scene in which the target robot is located, and the basic skill information of the target robot;

[0175] Using a pre-defined large language model, action sequence text and target direction are generated based on prompt words.

[0176] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0177] This application also provides an electronic device. Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 11 As shown, it includes: a processor 1101, a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device is running, the processor 1101 and the memory 1102 communicate via the bus 1103, and the machine-readable instructions are executed by the processor 1101 to perform the steps of the above method.

[0178] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0181] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method of training a robot control model, characterized by, The method comprises: acquiring a preset action data set, wherein the action data set comprises: a sample natural language command, and a preset reference action corresponding to the sample natural language command; using a preset large language model in a robot control model to generate action sequence text and a predicted target direction corresponding to the sample natural language command based on the sample natural language command; using an initial text encoder in the robot control model to encode the action sequence text to obtain action encoding corresponding to the sample natural language command; determining target action encoding corresponding to the action encoding from a preset action codebook according to the action encoding, wherein a plurality of preset action encodings are pre-stored in the preset action codebook; the initial action planning model in the robot control model comprises: a skill encoder, an initial strategy network, and a discriminator; using the skill encoder to perform skill encoding on the target action encoding to obtain an action vector corresponding to the sample natural language command; using the initial strategy network to generate action strategy information corresponding to the sample natural language command according to the predicted target direction and the action vector; using the discriminator to calculate an action reward parameter under the sample natural language command according to the preset reference action and action state information of a target robot under the action strategy information; acquiring an actual action direction of the target robot under the action strategy information; wherein the actual action direction comprises direction vectors of a pelvis, a left hip bone, and a right hip bone of an actual robot; calculating a task reward parameter under the sample natural language command according to the predicted target direction and the actual action direction; wherein the task reward parameter is a sum of a pelvis reward function, a left hip bone reward function, and a right hip bone reward function; performing parameter training on the initial strategy network according to the action reward parameter and the task reward parameter.

2. The method of claim 1, wherein, The method further comprises: calculating an encoding loss parameter according to the action encoding corresponding to the sample natural language command and the corresponding target action encoding; performing parameter training on the initial text encoder according to the encoding loss parameter.

3. The method of claim 1, wherein, The determining of the target action encoding corresponding to the action encoding from the preset action codebook comprises: calculating the similarity between each preset action encoding in the preset action codebook and the action encoding respectively according to the action encoding; determining the preset action encoding closest to the action encoding from the preset action codebook as the target action encoding according to the similarity.

4. The method of claim 1, wherein, The using of the preset large language model in the robot control model to generate the action sequence text and the predicted target direction corresponding to the sample natural language command based on the sample natural language command comprises: generating prompt words according to the sample natural language command, description information of the target robot, scene description information in which the target robot is located, and basic skill information of the target robot; using the preset large language model to generate the action sequence text and the predicted target direction based on the prompt words.

5. A robot control method characterized by, The method comprises: acquiring an input natural language command; adopting a preset large language model in a robot control model, generating action sequence text and a predicted target direction corresponding to the natural language command based on the natural language command; adopting an initial text encoder in the robot control model to encode the action sequence text to obtain action encoding corresponding to the natural language command; determining target action encoding corresponding to the action encoding from a preset action codebook according to the action encoding, wherein the preset action codebook pre-stores a plurality of preset action encodings; adopting a target action planning model in the robot control model to perform action planning according to the target action encoding and the predicted target direction, and generating action strategy information corresponding to the sample natural language command; wherein the robot control model is a model trained by the training method of the robot control model in any one of claims 1-4; controlling the robot according to the predicted action strategy information.

6. A device for training a robot control model, characterized by, The device comprises: an acquisition module configured to acquire a preset action dataset, wherein the action dataset comprises: a sample natural language command, and preset reference actions corresponding to the sample natural language command; a generation module configured to adopt a preset large language model in a robot control model, and generate action sequence text and a predicted target direction corresponding to the sample natural language command based on the sample natural language command; an encoding module configured to adopt an initial text encoder in the robot control model to encode the action sequence text to obtain action encoding corresponding to the sample natural language command; a determination module configured to determine target action encoding corresponding to the action encoding from a preset action codebook according to the action encoding, wherein the preset action codebook pre-stores a plurality of preset action encodings; the initial action planning model in the robot control model comprises: a skill encoder, an initial strategy network, and a discriminator; and a planning module is specifically configured to adopt the skill encoder to perform skill encoding on the target action encoding to obtain an action vector corresponding to the sample natural language command; adopting the initial strategy network, and generating action strategy information corresponding to the sample natural language command according to the predicted target direction and the action vector; a training module is specifically configured to adopt the discriminator, and calculate an action reward parameter under the sample natural language command according to the preset reference actions and action state information of a target robot under the action strategy information; acquire an actual action direction of the target robot under the action strategy information; wherein the actual action direction comprises direction vectors of a pelvis, a left hip bone, and a right hip bone of an actual robot; calculate a task reward parameter under the sample natural language command according to the predicted target direction and the actual action direction; wherein the task reward parameter is a sum of a pelvis reward function, a left hip bone reward function, and a right hip bone reward function; perform parameter training on the initial strategy network according to the action reward parameter and the task reward parameter.

7. An electronic device, comprising: comprises: A processor and a memory, the memory storing machine readable instructions executable by the processor, the processor executing the machine readable instructions to perform the steps of the method of any one of claims 1 to 5 when the electronic device is running.

8. A computer-readable storage medium, characterized in that, A computer readable storage medium storing a computer program, the computer program performing the steps of the method of any one of claims 1 to 5 when executed by a processor.

Images