Control instruction generation method and device, electronic equipment and storage medium

Abstract

The invention provides a control instruction generation method and device, electronic equipment and a storage medium, and belongs to the technical field of robotics.The method comprises the steps that related data of a robot control task are obtained, and the related data of the robot control task comprise a natural language instruction, a robot control instruction and a task execution result; based on a natural language instruction, a robot control instruction and a task execution result in the related data, model parameters of a basic code generation model are adjusted, an adjusted code generation model is obtained, and the basic code generation model is a code generation model used for generating logic codes; and when a natural language instruction sent by a user is received, generating a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model. By means of the method and device, the accuracy of robot control instruction generation and the accuracy of code generation are improved, and therefore execution of robot control tasks is more efficiently supported.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device, electronic device, and storage medium for generating control instructions. Background Art

[0002] With the continuous development of robotics technology, robots are increasingly being used in various fields. When robots perform tasks in different fields, users can issue task instructions to the robots. These instructions can be natural language tasks. The robots then interpret the natural language tasks issued by the users, obtain corresponding control instructions, and then perform the corresponding operations based on these control instructions.

[0003] In related technologies, robots are equipped with large language models, which are typically specially trained to process and generate programming-related text. These large language models understand the natural language tasks issued by users and convert them into computer program code, allowing the robot to execute the corresponding action plan, thereby achieving the natural language tasks issued by users.

[0004] In the above-mentioned related technologies, the large language model deployed in the robot is usually trained with a large amount of natural language text and a small amount of code data, which fails to fully utilize the similarities between the two tasks of control instruction generation and code generation. Therefore, the accuracy of the generated control instructions is low, which ultimately causes the problem of inaccurate control of the robot. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, electronic device and storage medium for generating control instructions, aiming to solve the problem of inaccurate control of traditional robots.

[0006] A first aspect of an embodiment of the present application provides a method for generating a control instruction, the method comprising:

[0007] Acquiring relevant data of the robot control task, wherein the relevant data of the robot control task includes natural language instructions, robot control instructions and task execution results;

[0008] Adjusting model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code;

[0009] When a natural language instruction issued by a user is received, a robot control instruction corresponding to the natural language instruction is generated based on the adjusted code generation model.

[0010] In some embodiments, adjusting the model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain the adjusted code generation model includes:

[0011] Annotating the natural language instructions, robot control instructions, and task execution results in the relevant data of the robot control task to obtain training samples;

[0012] The model parameters of the basic code generation model are adjusted based on the training sample to obtain the adjusted code generation model.

[0013] In some embodiments, adjusting the model parameters of the basic code generation model based on the training sample to obtain the adjusted code generation model includes:

[0014] Extracting a current sample natural language instruction from the training sample, inputting the sample natural language instruction into the code generation model, and the code generation model generating an actual robot control instruction corresponding to the sample natural language instruction using current model parameters;

[0015] Obtaining an actual task execution result obtained by the robot executing the actual robot control instruction;

[0016] generating a feedback signal based on a sample task execution result and an actual task execution result corresponding to the sample natural language instruction;

[0017] determining a loss value between an actual robot control instruction corresponding to the sample natural language instruction and the sample robot control instruction;

[0018] adjusting model parameters of the code generation model based on the feedback signal and the loss value;

[0019] Based on the adjusted parameters, the steps of extracting the current sample natural language instruction from the training sample and subsequent steps are executed until the model converges.

[0020] In some embodiments, adjusting the model parameters of the code generation model based on the feedback signal and the loss value includes:

[0021] generating a fine-tuning parameter based on the feedback signal and the loss value;

[0022] Based on preset weights, the fine-tuning parameters and the original model parameters of the code generation model are weighted to obtain adjusted model parameters.

[0023] In some embodiments, before adjusting the model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain the adjusted code generation model, the method further includes:

[0024] Selecting an optimal code generation model from multiple code generation models to be selected based on a preset selection criterion;

[0025] Model training is performed on the optimal code generation model based on the training samples to obtain the adjusted code generation model.

[0026] In some embodiments, obtaining data related to the robot control task includes:

[0027] Determining the type of tasks that the robot can perform;

[0028] Obtain relevant data of the robot control task according to the task type.

[0029] In some embodiments, the relevant data further includes contextual features of the robot performing the task;

[0030] The adjusting of model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model includes:

[0031] The current sample natural language instructions and sample context features are extracted from the training sample, and the sample natural language instructions and the sample context features are input into the code generation model. The code generation model generates the actual robot control instructions corresponding to the sample natural language instructions under the sample context features through the current model parameters.

[0032] A second aspect of an embodiment of the present application provides a device for generating a control instruction, the device comprising:

[0033] An acquisition unit, configured to acquire data related to a robot control task, wherein the data related to the robot control task includes natural language instructions, robot control instructions, and task execution results;

[0034] an adjustment unit, configured to adjust model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code;

[0035] The generating unit is configured to generate robot control instructions corresponding to the natural language instructions based on the adjusted code generation model when receiving the natural language instructions issued by the user.

[0036] In some embodiments, the adjusting unit is configured to perform data labeling on the natural language instructions, the robot control instructions, and the task execution results in the related data of the robot control task to obtain training samples.

[0037] The model parameters of the basic code generation model are adjusted based on the training samples to obtain the adjusted code generation model.

[0038] In some embodiments, the adjusting unit is configured to extract a current sample natural language instruction from the training samples, input the sample natural language instruction into the code generation model, and generate actual robot control instructions corresponding to the sample natural language instruction by the code generation model through current model parameters; obtain actual task execution results obtained by the robot executing the actual robot control instructions; generate a feedback signal based on a sample task execution result corresponding to the sample natural language instruction and the actual task execution result; determine a loss value between the actual robot control instructions corresponding to the sample natural language instruction and the sample robot control instructions; adjust the model parameters of the code generation model based on the feedback signal and the loss value; and perform the steps of extracting a current sample natural language instruction from the training samples and the like based on the adjusted parameters until the model converges.

[0039] In some embodiments, the adjusting unit is configured to generate fine-tuning parameters based on the feedback signal and the loss value; and weight the fine-tuning parameters and original model parameters of the code generation model based on a preset weight to obtain adjusted model parameters.

[0040] In some embodiments, the apparatus further comprises:

[0041] The selecting unit is configured to select an optimal code generation model from the plurality of code generation models to be selected based on a preset selection standard.

[0042] The adjusting unit is configured to perform model training on the optimal code generation model based on the training samples to obtain the adjusted code generation model.

[0043] In some embodiments, the obtaining unit is configured to determine a task type of the task executable by the robot; and obtain the related data of the robot control task according to the task type.

[0044] In some embodiments, the related data further comprises context features of the robot executing the task.

[0045] The adjustment unit is used to extract the current sample natural language instructions and sample context features from the training sample, and input the sample natural language instructions and the sample context features into the code generation model. The code generation model generates the actual robot control instructions corresponding to the sample natural language instructions under the sample context features through the current model parameters.

[0046] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the control instruction generation method described above when executing the computer program.

[0047] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a control instruction as described above is implemented.

[0048] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the control instruction generation method described above.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] In an embodiment of the present application, the model parameters of the basic code generation model are adjusted by the natural language instructions, robot control instructions and task execution results in the relevant data of the robot control task to obtain an adjusted code generation model. When the natural language instructions issued by the user are received, the robot control instructions corresponding to the natural language instructions are generated based on the adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code. In this way, the model parameters of the basic code generation model for generating logic code are adjusted by the natural language instructions, robot control instructions and task execution results. The similarity between the two tasks of generating robot control instructions and generating code is combined, and the rich world knowledge and reasoning ability contained in the large code generation model trained with massive corpus is utilized, thereby further improving the accuracy of robot control instruction generation and the accuracy of code generation, so as to focus on improving the accuracy and applicability of robot embodied instruction generation, thereby more efficiently supporting the execution of robot control tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic flow chart of a method for generating a control instruction provided by an exemplary embodiment is shown;

[0052] Figure 2A schematic flow chart of a method for generating a control instruction provided by an exemplary embodiment is shown;

[0053] Figure 3 A schematic structural diagram of a device for generating a control instruction provided by an exemplary embodiment is shown;

[0054] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the technical problems, technical solutions and beneficial effects to be solved by this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0057] With the continuous development of robotics technology, robots are increasingly being used in various fields. When robots perform tasks in different fields, users can issue task instructions to the robots. These instructions can be natural language tasks. The robots then interpret the natural language tasks issued by the users, obtain corresponding control instructions, and then perform the corresponding operations based on these control instructions.

[0058] In some embodiments, robots are equipped with large language models, which are typically specially trained to process and generate programming-related text. These large language models understand the natural language tasks issued by users and convert them into computer program code, allowing the robot to execute the corresponding action plan, thereby achieving the natural language tasks issued by users.

[0059] In the above embodiments, the large language model deployed in the robot is usually trained using a large amount of natural language text and a small amount of code data, which fails to fully utilize the similarities between the two tasks of control instruction generation and code generation. Therefore, the accuracy of the generated control instructions is low, which ultimately causes the problem of inaccurate control of the robot.

[0060] In order to make the model deployed in the robot more suitable for the robot to generate control instructions, the application provides a method and device for generating control instructions, a robot and a storage medium. The model parameters of a basic code generation model are adjusted through natural language instructions, robot control instructions and task execution results in the related data of the robot control task, to obtain an adjusted code generation model. When a natural language instruction issued by a user is received, the robot control instruction corresponding to the natural language instruction is generated based on the adjusted code generation model. The basic code generation model is a code generation model for generating logical code. Thus, the model parameters of the basic code generation model for generating logical code are adjusted through natural language instructions, robot control instructions and task execution results, the similarity of the two tasks of generating robot control instructions and generating code is combined, the rich world knowledge and reasoning ability contained in the training of the code generation large model using a large amount of corpus are utilized, and the accuracy of robot control instruction generation and the accuracy of code generation are further improved. The accuracy and applicability of robot embodiment instruction generation are focused on to improve, thereby more efficiently supporting the execution of the robot control task.

[0061] The application will be described in detail below with reference to specific embodiments. Referring to Figure 1 which shows a flowchart of a method for generating control instructions provided by an exemplary embodiment, which is taken as an example but not as a limitation. The method is performed by an electronic device, which can communicate with a robot. The electronic device can be a processing unit in the robot, or the electronic device can be an electronic device independent of the robot. In the embodiments of the application, this is not specifically limited.

[0062] In S101, the electronic device obtains related data of a robot control task, which includes natural language instructions, robot control instructions and task execution results.

[0063] The related data of the robot control task is the related data recorded when the robot actually executes the task. The related task includes actual execution records, task execution data in a simulated environment and public control task data sets, etc. The related data of the robot control task can be the related data generated by the robot itself when the robot executes the task, and the related data of the robot control task can also be the related data generated by other robots when the robots execute the task. In the embodiments of the application, this is not specifically limited.

[0064] It should be noted that the robot control task refers to robot embodiment instructions, including tasks executed in multiple application scenarios. The application scenarios include but are not limited to industrial robot pipeline tasks, service robot and human interaction tasks, robot autonomous navigation tasks, etc.

[0065] The electronic device can obtain the required data related to the robot control task from the relevant database. The robot can also receive the data related to the robot control task sent by the relevant technician through other electronic devices. In the embodiments of the present application, this is not specifically limited.

[0066] In some embodiments, when a robot performs a task, it obtains data related to the robot's control task by using sensors and the robot's operation log to record the mapping relationship between control instructions and environmental conditions. Sensors include, but are not limited to, cameras, radars, and force sensors.

[0067] To ensure that the adjusted code generation model is suitable for the different types of tasks performed by the robot, the electronic device can also obtain relevant data based on different task types in this step. Accordingly, the electronic device determines the task type that the robot can perform and obtains relevant data for the robot control task based on the task type.

[0068] In other embodiments, after the electronic device obtains the data related to the robot control task, it pre-processes the data related to the robot control task to ensure that the data content meets the task instruction requirements. The pre-processing of the data related to the robot control task can be achieved by the following steps S1011-S1013, including:

[0069] S1011, the electronic device performs deduplication and formatting processing on the data related to the robot control task.

[0070] In this step, the electronic device identifies duplicate instruction records from the relevant data of the robot control task, removes the duplicate instruction records, and converts the relevant data into a unified data format. The data format can be set as needed and is not specifically limited in the embodiments of the present application. For example, the data format can be JSON, YAML, or other formats.

[0071] S1012, the electronic device filters abnormal data from the relevant data of the robot control task.

[0072] Electronic devices automatically detect and eliminate relevant data with anomalies, such as instructions that do not match environmental conditions or data that is incorrectly labeled.

[0073] S1013, the electronic device performs noise filtering on the data related to the robot control task.

[0074] The electronic device can filter irrelevant or redundant information in the relevant data of the robot control task through a natural language processing (NLP) tool and rules to ensure that the content of the relevant data is data related to the robot control task.

[0075] It should be noted that the electronic device can perform at least one of the above steps S1011-S1013, and the order of the preprocessing operation of the electronic device is not limited in the present application. The electronic device can also select other preprocessing methods to preprocess the relevant data of the robot control task, and the method of preprocessing the relevant data of the robot control task by the electronic device is not limited in the embodiments of the present application.

[0076] In the present implementation, the relevant data of the robot control task is preprocessed to meet the requirements of the robot for the format, content, etc. of the relevant data, thereby improving the efficiency of subsequent processing of the relevant data.

[0077] S102, the electronic device adjusts the model parameters of the basic code generation model based on the natural language instructions, robot control instructions and task execution results in the relevant data, to obtain an adjusted code generation model.

[0078] In some embodiments, the basic code generation model is an open source code generation model, such as DeepSeek-Coder, Qwen-Coder, StarCoder2, etc.

[0079] In some embodiments, in order to ensure that the trained model can better meet the requirements of code generation, the electronic device selects an optimal code generation model from a plurality of code generation models to be selected based on a preset selection standard; and performs model training on the optimal code generation model based on the training sample to obtain the adjusted code generation model.

[0080] The selection standard can be set as needed, and the selection standard is not limited in the embodiments of the present application. For example, the selection standard can be a standard for evaluating the code generation ability of the model, for example, the selection standard includes the code understanding ability, code completion ability and logical consistency of the model, etc.

[0081] In the present implementation, the code generation model is preliminarily evaluated to select a basic code generation model that better meets the requirements of code generation.

[0082] It should be noted that the model parameter adjustment process (steps S101-S102) can also be performed by other electronic devices. Accordingly, the other electronic devices fine-tune the code generation model and send the fine-tuned code generation model to the electronic device currently communicating with the robot. The process of fine-tuning the code generation model by the other electronic devices is the same as the process of fine-tuning the code generation model by the current electronic device, and will not be repeated here.

[0083] S103, when receiving a natural language instruction issued by the user, the electronic device generates a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model.

[0084] When the robot receives a natural language instruction from a user, it generates a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model. The robot control instruction is a logical code instruction that the robot can recognize. Accordingly, the robot performs the corresponding task based on the control instruction.

[0085] In some embodiments, the robot communicates with the electronic device. Accordingly, when the robot receives a natural language instruction issued by the user, it sends the natural language instruction to the electronic device. Accordingly, the electronic device generates a corresponding robot control instruction based on the natural language instruction, and sends the robot control instruction to the robot. The robot performs the corresponding task based on the control instruction sent by the electronic device.

[0086] In some embodiments, after the electronic device generates the adjusted code generation model through steps S101-S102, it transmits the relevant parameters of the adjusted code generation model to the robot. The robot then deploys the adjusted code generation model based on the relevant parameters of the code generation model transmitted by the electronic device. Accordingly, when the robot receives a natural language instruction from a user, it generates a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model. The robot control instruction is a logical code instruction recognizable by the robot. Accordingly, the robot executes the corresponding task based on the control instruction.

[0087] In an embodiment of the present application, the model parameters of the basic code generation model are adjusted by the natural language instructions, robot control instructions and task execution results in the relevant data of the robot control task to obtain an adjusted code generation model. When the natural language instructions issued by the user are received, the robot control instructions corresponding to the natural language instructions are generated based on the adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code. In this way, the model parameters of the basic code generation model for generating logic code are adjusted by the natural language instructions, robot control instructions and task execution results. The similarity between the two tasks of generating robot control instructions and generating code is combined, and the rich world knowledge and reasoning ability contained in the large code generation model trained with massive corpus is utilized, thereby further improving the accuracy of robot control instruction generation and the accuracy of code generation, so as to focus on improving the accuracy and applicability of robot embodied instruction generation, thereby more efficiently supporting the execution of robot control tasks.

[0088] In some embodiments, the electronic device labels the acquired data related to the robot control task to obtain training samples, and then performs model training on the basic code generation model based on the training samples. Figure 2 , which shows a flow chart of a method for training a basic code generation model in a control instruction generation method provided by an exemplary embodiment. By way of example and not limitation, the method is performed by an electronic device that can communicate with a robot. The electronic device can be a processing unit in the robot, or it can be an electronic device independent of the robot. This is not specifically limited in the embodiments of the present application.

[0089] S201, the electronic device obtains relevant data of a robot control task, where the relevant data of the robot control task includes natural language instructions, robot control instructions, and task execution results.

[0090] The principle of this step is the same as that of step S101 and will not be repeated here.

[0091] S202 , the electronic device performs data annotation on the natural language instructions, robot control instructions, and task execution results in the relevant data of the robot control task to obtain training samples.

[0092] A neural network model for data labeling can be deployed in the electronic device. The neural network model is used to label the natural language instructions, robot control instructions and task execution results in the relevant data of the robot control task to obtain training samples.

[0093] In this step, the electronic device annotates the data related to the robot control task into a triplet of "natural language command - robot control command - execution result." The natural language command is the command issued by the robot user, such as "grab the cup in front of the right." The robot control command is the code form converted from the command to the underlying control logic, such as "move_to(x, y, z) → grasp(object_id)." The execution result is the actual status of the robot after execution, such as success, failure, or the degree of deviation.

[0094] In some embodiments, the data related to the robot control task also includes contextual features of the robot performing the task. Accordingly, the robot can also annotate these contextual features, which include robot position, sensor data, hardware parameters, and so on. Adding contextual features to the data related to the robot control task enriches the data dimension of the training samples and makes the code generation model more accurate.

[0095] In some embodiments, after the robot labels the relevant data of the robot control task, the labeled data can also be manually verified to ensure that the labeled training samples are accurate.

[0096] To increase data diversity, data augmentation techniques are used to generate diverse instruction data. For example, multiple expressions can be designed for the same task, such as "grab the cup" and "pick up the cup." Alternatively, expressions in other languages ​​can be added. This is not specifically limited in the present embodiment. Furthermore, to improve the coverage of the dataset, multiple task scenarios can be simulated.

[0097] S203: The electronic device adjusts the model parameters of the basic code generation model based on the training sample to obtain the adjusted code generation model.

[0098] In this step, the basic code generation model is fine-tuned using the annotated natural language instructions, robot control instructions, and task execution results so that the basic code generation model can fit the application scenario of the robot. The process of fine-tuning the basic code generation model can be achieved through the following steps S2031-S2036, including:

[0099] S2031, the electronic device extracts the current sample natural language instruction from the training sample, and inputs the sample natural language instruction into the code generation model. The code generation model generates the actual robot control instruction corresponding to the sample natural language instruction through the current model parameters.

[0100] The electronic device extracts any training sample from the training samples, and inputs the sample natural language instructions marked in the training sample into the code generation model. Accordingly, the code generation model generates the actual robot control instructions corresponding to the sample natural language instructions through the current model parameters.

[0101] S2032, the electronic device obtains the actual task execution result obtained by the robot executing the actual robot control instruction.

[0102] In some embodiments, the electronic device simulates the actual task execution result corresponding to the actual robot control instruction. In some embodiments, the electronic device sends the actual robot control instruction to the robot, and the robot executes the actual robot control instruction to obtain the actual task execution result. The electronic device obtains the actual task execution result after the robot executes the actual robot control instruction.

[0103] S2033: The electronic device generates a feedback signal based on the sample task execution result and the actual task execution result corresponding to the sample natural language instruction.

[0104] When the sample task execution result corresponding to the natural language instruction matches the actual task execution result, the electronic device generates a positive feedback signal. When the sample task execution result corresponding to the natural language instruction does not match the actual task execution result, the electronic device generates a negative feedback signal.

[0105] The matching of the sample task execution result and the actual task execution result means that the sample task execution result and the actual task execution result are the same, or the difference between the sample task execution result and the actual task execution result is within a preset difference range.

[0106] S2034: The electronic device determines a loss value between the actual robot control instruction corresponding to the sample natural language instruction and the sample robot control instruction.

[0107] The electronic device extracts features of the actual robot control command and the sample robot control command, compares the features between the two, and calculates a loss value. The method by which the electronic device calculates the loss value can be configured as needed. In the embodiments of the present application, the method for calculating the loss value is not specifically limited. For example, the electronic device can calculate the loss value using a cross-entropy loss algorithm.

[0108] S2035: The electronic device adjusts the model parameters of the code generation model based on the feedback signal and the loss value.

[0109] In this step, the electronic device determines a model adjustment coefficient based on the feedback signal and adjusts the model parameters based on the loss value. To ensure the generalization capability of the code generation model, a hybrid weighting technique can be used to adjust the parameters of the code generation model in this step. This process includes: the electronic device generates fine-tuning parameters based on the feedback signal and the loss value; and weights the fine-tuning parameters and the original model parameters of the code generation model based on preset weights to obtain adjusted model parameters.

[0110] S2036, the electronic device executes the natural language instruction of extracting the current sample from the training sample and subsequent steps based on the adjusted parameters until the model converges.

[0111] The electronic device cyclically executes steps S2031-S2035 to adjust the code generation model until the model converges. The model convergence can be when the model training coefficient reaches a preset number of times, or the model is evaluated using multiple indicators. For example, it can be determined whether the code generated by the adjusted code generation model conforms to the expected logic, or whether the actual execution effect of the generated code failure meets the expected effect, or the adaptability of the adjusted code generation model to uninvolved scenarios or instructions.

[0112] When adjusting the model parameters of the code generation model, the electronic device considers factors such as the learning rate and training batch size to ensure efficient model convergence. This training process can be performed in the robot's real or simulated environment, utilizing execution feedback data to optimize the model. Iterative training is performed using a policy optimization algorithm (such as Proximal Policy Optimization (PPO)).

[0113] In some embodiments, in order to ensure that model training can meet each application scenario, before model training, training samples can also be divided according to task type, for example, classified into navigation instruction data, grasping task instruction data, environment modeling instruction data, etc.

[0114] In addition, to prevent model overfitting, it is necessary to ensure the consistency of data distribution among the training set, validation set, and test set. This consistency refers to the consistency of the type and number of training samples.

[0115] In other embodiments, the electronic device may further train the code generation model based on the contextual features of the robot performing the task. Accordingly, the relevant data also includes the contextual features of the robot performing the task. The model training process includes extracting the current sample natural language instruction and sample contextual features from the training sample, inputting the sample natural language instruction and the sample contextual features into the code generation model, and the code generation model generating the actual robot control instruction corresponding to the sample natural language instruction under the sample contextual features using the current model parameters.

[0116] The training process has the same principles as those of steps S2031-S2036 above, and will not be described again here.

[0117] S204 , when receiving a natural language instruction from the user, the robot generates a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model.

[0118] The principle of this step is the same as that of step S103 and will not be repeated here.

[0119] In an embodiment of the present application, the model parameters of the basic code generation model are adjusted by the natural language instructions, robot control instructions and task execution results in the relevant data of the robot control task to obtain an adjusted code generation model. When the natural language instructions issued by the user are received, the robot control instructions corresponding to the natural language instructions are generated based on the adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code. In this way, the model parameters of the basic code generation model for generating logic code are adjusted by the natural language instructions, robot control instructions and task execution results. The similarity between the two tasks of generating robot control instructions and generating code is combined, and the rich world knowledge and reasoning ability contained in the large code generation model trained with massive corpus is utilized, thereby further improving the accuracy of robot control instruction generation and the accuracy of code generation, so as to focus on improving the accuracy and applicability of robot embodied instruction generation, thereby more efficiently supporting the execution of robot control tasks.

[0120] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] See also Figure 3 , which shows a schematic diagram of the structure of a control instruction generation device provided by the present application, including various units for executing various steps in the above embodiment, see Figure 3 , the control instruction generating device includes:

[0122] An acquisition unit 301 is configured to acquire data related to a robot control task, where the data includes natural language instructions, robot control instructions, and task execution results.

[0123] An adjustment unit 302 is configured to adjust model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, where the basic code generation model is a code generation model for generating logic code;

[0124] The generating unit 303 is configured to generate a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model when receiving a natural language instruction issued by the user.

[0125] In some embodiments, the adjustment unit 302 is configured to perform data annotation on natural language instructions, robot control instructions, and task execution results in the relevant data of the robot control task to obtain training samples;

[0126] The model parameters of the basic code generation model are adjusted based on the training sample to obtain the adjusted code generation model.

[0127] In some embodiments, the adjustment unit 302 is used to extract the current sample natural language instruction from the training sample, input the sample natural language instruction into the code generation model, and the code generation model generates the actual robot control instruction corresponding to the sample natural language instruction through the current model parameters; obtains the actual task execution result obtained by the robot executing the actual robot control instruction; generates a feedback signal based on the sample task execution result and the actual task execution result corresponding to the sample natural language instruction; determines the loss value between the actual robot control instruction and the sample robot control instruction corresponding to the sample natural language instruction; adjusts the model parameters of the code generation model based on the feedback signal and the loss value; and executes the steps of extracting the current sample natural language instruction from the training sample and subsequent steps based on the adjusted parameters until the model converges.

[0128] In some embodiments, the adjustment unit 302 is configured to generate a fine-tuning parameter based on the feedback signal and the loss value; and weight the fine-tuning parameter and the original model parameter of the code generation model based on a preset weight to obtain an adjusted model parameter.

[0129] In some embodiments, the apparatus further comprises:

[0130] A selection unit, configured to select an optimal code generation model from a plurality of code generation models to be selected based on a preset selection criterion;

[0131] The adjusting unit 302 is configured to perform model training on the optimal code generation model based on the training sample to obtain the adjusted code generation model.

[0132] In some embodiments, the acquisition unit 301 is used to determine the task type of the robot executable task; and acquire relevant data of the robot control task according to the task type.

[0133] In some embodiments, the relevant data also includes contextual features of the robot performing the task;

[0134] The adjustment unit 302 is used to extract the current sample natural language instructions and sample context features from the training sample, and input the sample natural language instructions and the sample context features into the code generation model. The code generation model generates the actual robot control instructions corresponding to the sample natural language instructions under the sample context features through the current model parameters.

[0135] In an embodiment of the present application, the model parameters of the basic code generation model are adjusted by the natural language instructions, robot control instructions and task execution results in the relevant data of the robot control task to obtain an adjusted code generation model. When the natural language instructions issued by the user are received, the robot control instructions corresponding to the natural language instructions are generated based on the adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code. In this way, the model parameters of the basic code generation model for generating logic code are adjusted by the natural language instructions, robot control instructions and task execution results. The similarity between the two tasks of generating robot control instructions and generating code is combined, and the rich world knowledge and reasoning ability contained in the large code generation model trained with massive corpus is utilized, thereby further improving the accuracy of robot control instruction generation and the accuracy of code generation, so as to focus on improving the accuracy and applicability of robot embodied instruction generation, thereby more efficiently supporting the execution of robot control tasks.

[0136] Figure 4 FIG. 1 is a schematic diagram of an electronic device provided by an exemplary embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40, such as a control instruction generation program. When the processor 40 executes the computer program 42, the steps of the above-mentioned control instruction generation method embodiments are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 3 Functions of units 301 to 303 are shown.

[0137] Exemplarily, the computer program 42 may be divided into one or more units, which are stored in the memory 41 and executed by the processor 40 to complete the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 42 in the electronic device 4. For example, the computer program 42 may be divided into an acquisition unit, an adjustment unit, and a generation unit, and the specific functions of each module are as follows:

[0138] An acquisition unit 301 is configured to acquire data related to a robot control task, where the data includes natural language instructions, robot control instructions, and task execution results.

[0139] An adjustment unit 302 is configured to adjust model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, where the basic code generation model is a code generation model for generating logic code;

[0140] The generating unit 303 is configured to generate a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model when receiving a natural language instruction issued by the user.

[0141] The electronic device 4 may be any electronic device with a control function. The electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 4 may also include input and output devices, network access devices, buses, etc.

[0142] The processor 40 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0143] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Furthermore, the memory 41 can also include both the internal storage unit of the electronic device 4 and an external storage device. The memory 41 is used to store the computer program and other programs and data required by the terminal device. The memory 41 can also be used to temporarily store data that has been output or is about to be output.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0145] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0150] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0151] The embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] The embodiments of the present application further provide a computer program product, which, when executed on a mobile terminal, enables the mobile terminal to implement the steps of the above-mentioned method embodiments.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for generating a control instruction, characterized in that: The method comprises: Acquiring relevant data of the robot control task, wherein the relevant data of the robot control task includes natural language instructions, robot control instructions and task execution results; Adjusting model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code; When a natural language instruction issued by a user is received, a robot control instruction corresponding to the natural language instruction is generated based on the adjusted code generation model.

2. The method according to claim 1, wherein The adjusting of model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model includes: Annotating the natural language instructions, robot control instructions, and task execution results in the relevant data of the robot control task to obtain training samples; The model parameters of the basic code generation model are adjusted based on the training sample to obtain the adjusted code generation model.

3. The method according to claim 2, wherein The adjusting the model parameters of the basic code generation model based on the training sample to obtain the adjusted code generation model includes: Extracting a current sample natural language instruction from the training sample, inputting the sample natural language instruction into the code generation model, and the code generation model generating an actual robot control instruction corresponding to the sample natural language instruction using current model parameters; Obtaining an actual task execution result obtained by the robot executing the actual robot control instruction; generating a feedback signal based on a sample task execution result and an actual task execution result corresponding to the sample natural language instruction; determining a loss value between an actual robot control instruction corresponding to the sample natural language instruction and the sample robot control instruction; adjusting model parameters of the code generation model based on the feedback signal and the loss value; Based on the adjusted parameters, the steps of extracting the current sample natural language instruction from the training sample and subsequent steps are executed until the model converges.

4. The method according to claim 3, wherein The adjusting the model parameters of the code generation model based on the feedback signal and the loss value includes: generating a fine-tuning parameter based on the feedback signal and the loss value; Based on preset weights, the fine-tuning parameters and the original model parameters of the code generation model are weighted to obtain adjusted model parameters.

5. The method according to claim 1, wherein Before adjusting the model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain the adjusted code generation model, the method further includes: Selecting an optimal code generation model from multiple code generation models to be selected based on a preset selection criterion; Model training is performed on the optimal code generation model based on the training samples to obtain the adjusted code generation model.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of the relevant data of the robot control task includes: Determining the type of tasks that the robot can perform; Obtain relevant data of the robot control task according to the task type.

7. The method according to any one of claims 1 to 5, wherein: The relevant data also includes contextual features of the robot performing tasks; The adjusting of model parameters of the basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model includes: The current sample natural language instructions and sample context features are extracted from the training sample, and the sample natural language instructions and the sample context features are input into the code generation model. The code generation model generates the actual robot control instructions corresponding to the sample natural language instructions under the sample context features through the current model parameters.

8. A control instruction generating device, characterized in that: The device comprises: An acquisition unit, configured to acquire data related to a robot control task, wherein the data related to the robot control task includes natural language instructions, robot control instructions, and task execution results; an adjustment unit, configured to adjust model parameters of a basic code generation model based on the natural language instructions, robot control instructions, and task execution results in the relevant data to obtain an adjusted code generation model, wherein the basic code generation model is a code generation model for generating logic code; The generation unit is configured to, when receiving a natural language instruction issued by a user, generate a robot control instruction corresponding to the natural language instruction based on the adjusted code generation model.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a control instruction according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a control instruction according to any one of claims 1 to 7 is implemented.

Images