Robot motion control code generation method and system

Abstract

The embodiment of the invention provides a robot motion control code generation method and system, and belongs to the technical field of robots. The method comprises the steps of obtaining a natural language task instruction; analyzing the perception information of the robot according to the natural language task instruction to obtain a natural language plan; performing code generation according to natural language planning to obtain an initial robot motion control code; verifying the initial robot motion control code, and determining whether the initial robot motion control code has an error or not; and if the initial robot motion control code has an error, performing iterative correction on the initial robot motion control code to obtain a target robot motion control code. According to the embodiment of the invention, the reliability and accuracy of code generation can be improved, and higher-degree automatic code generation is realized.

Description

A method and system for generating robot motion control code Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and system for generating robot motion control code. Background Technology

[0002] Industrial robots play an irreplaceable role in fields such as automated production, significantly improving production efficiency and reducing labor costs. However, traditional robot programming methods, such as programming using KUKA Robot Language (KRL), typically require skilled technicians to manually write and debug the code, a complex and time-consuming process that limits the flexibility of robot applications in more unstructured and dynamically changing scenarios.

[0003] To achieve more efficient human-computer interaction, the industry has begun exploring the use of natural language commands to control robots. However, there is currently no technology that can perfectly solve the end-to-end generation problem from natural language commands to reliable robot executable code, and a lot of manual debugging is still required, thus failing to achieve true automation.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a method and system for generating robot motion control code, which can improve the reliability and accuracy of the generated code and achieve a higher degree of automated code generation.

[0006] To achieve the above objectives, one aspect of this application proposes a method for generating robot motion control code. The method includes: acquiring natural language task instructions; parsing the robot's perception information according to the natural language task instructions to obtain natural language planning; generating code according to the natural language planning to obtain initial robot motion control code; verifying the initial robot motion control code to determine whether there are errors in the initial robot motion control code; and if there are errors in the initial robot motion control code, iteratively correcting the initial robot motion control code to obtain target robot motion control code.

[0007] In some embodiments, parsing the robot's perception information according to the natural language task instructions to obtain natural language planning includes: recognizing the natural language task instructions to determine the target object; calculating the coordinates of the target object in the perception information to obtain the world coordinates of the target object; and generating the natural language planning based on the natural language task instructions and the world coordinates.

[0008] In some embodiments, the perceived information includes a scene image, and the step of calculating the coordinates of the target object in the perceived information to obtain the world coordinates of the target object includes: performing image segmentation on the scene image to obtain the contour mask of each object; calculating the similarity of the image region corresponding to each contour mask according to the text description of the target object to determine the target mask of the target object; calculating the geometric center of the target mask to obtain the pixel coordinates of the target object; and performing coordinate transformation on the pixel coordinates to obtain the world coordinates.

[0009] In some embodiments, the step of generating initial robot motion control code based on the natural language planning includes: acquiring grammatical knowledge from a preset robot control language grammar knowledge base; and generating initial robot motion control code based on the grammatical knowledge and the natural language planning.

[0010] In some embodiments, verifying the initial robot motion control code includes: performing syntax verification on the initial robot motion control code and generating a first feedback result, wherein the first feedback result is used to iteratively correct the initial robot motion control code.

[0011] In some embodiments, if the initial robot motion control code contains errors, the initial robot motion control code is iteratively corrected to obtain the target robot motion control code. This includes: correcting the initial robot motion control code according to the code error type and code error location in the first feedback result to obtain intermediate corrected code; verifying the intermediate corrected code to obtain a second feedback result; if the second feedback result is empty, then using the intermediate corrected code as the target robot motion control code; if the second feedback result is not empty, then correcting the initial robot motion control code according to the code error type and code error location in the second feedback result.

[0012] To achieve the above objectives, another aspect of this application proposes a robot motion control code generation system. The system includes: an acquisition module for acquiring natural language task instructions; a parsing module for parsing the robot's perception information based on the natural language task instructions to obtain a natural language plan; a generation module for generating code based on the natural language plan to obtain initial robot motion control code; a verification module for verifying the initial robot motion control code to determine if errors exist; and a correction module for iteratively correcting the initial robot motion control code if errors exist, to obtain target robot motion control code.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for generating robot motion control code. This solution generates initial robot motion control code by combining natural language task instructions with perception information of the surrounding environment, and finally generates executable target robot motion control code through autonomous code correction. This significantly improves the reliability and accuracy of the generated code, reduces the complexity of robot programming, reduces the work of manually debugging code, and achieves a higher degree of automated code generation. Attached Figure Description

[0017] Figure 1 is a flowchart of a robot motion control code generation method provided in an embodiment of this application; Figure 2 is a schematic diagram of a two-layer agent collaboration framework provided in an embodiment of this application; Figure 3 is a flowchart of the first agent operation provided in an embodiment of this application; Figure 4 is a flowchart of the second agent operation provided in an embodiment of this application; Figure 5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Industrial robots play an irreplaceable role in fields such as automated production, significantly improving production efficiency and reducing labor costs. However, traditional robot programming methods, such as programming using KUKA Robot Language (KRL), typically require skilled technicians to manually write and debug the code, a complex and time-consuming process that limits the flexibility of robot applications in more unstructured and dynamically changing scenarios.

[0021] To achieve more efficient human-computer interaction, the industry has begun to explore using natural language commands to control robots. This approach requires translating high-level, ambiguous natural language tasks (such as "grab an apple from the table") into low-level control code that the robot can execute, containing precise coordinates and action sequences.

[0022] Currently, there is no technology that can perfectly solve the technical problem of end-to-end generation from natural language instructions to reliable robot executable code.

[0023] Additionally, some related technologies mention using a single large language model to directly translate user commands into robot control code. The model receives natural language input and attempts to output the complete program in one go.

[0024] However, the above methods have the following shortcomings: A disconnect between perception and planning: a single model lacks a mechanism for interacting with the physical world and cannot actively acquire the precise position, orientation, and other geometric information of objects mentioned in the instructions. The generated code is often based on guesses or preset values, making it difficult to adapt to real, dynamic operating environments, leading to inaccurate positioning and task failure.

[0025] Unreliable code syntax: While large language models can mimic grammatical structures when generating code, they often produce syntax errors that do not conform to the strict specifications of specific robot languages ​​(such as KRL). Due to the lack of effective verification and correction mechanisms, the generated code often cannot be run directly on the robot controller and still requires a lot of manual debugging, failing to achieve true automation.

[0026] In view of this, this application provides a method and system for generating robot motion control code. This method generates initial robot motion control code by combining natural language task instructions with perception information of the surrounding environment, and finally generates executable target robot motion control code through autonomous code correction. This significantly improves the reliability of the generated code, reduces the complexity of robot programming, reduces the work of manually debugging code, and achieves a higher degree of automation.

[0027] The robot motion control code generation method provided in this application relates to the field of robotics technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the robot motion control code generation method, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] Figure 1 is an optional flowchart of a robot motion control code generation method provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S101 to S105.

[0030] Step S101: Obtain natural language task instructions; Step S102: Parse the robot's perception information according to the natural language task instructions to obtain natural language planning; Step S103: Generate code according to the natural language planning to obtain initial robot motion control code; Step S104: Verify the initial robot motion control code to determine if there are any errors; Step S105: If there are errors in the initial robot motion control code, iteratively correct the initial robot motion control code to obtain the target robot motion control code.

[0031] Steps S101 to S105, as illustrated in this embodiment, parse the perceived information using natural language task instructions, breaking down the barrier between the language model and the physical world. This allows the generated robot motion trajectory to accurately correspond to objects in the real world, improving the accuracy of task execution. Through a code correction mechanism, the code generation process is endowed with self-correcting capabilities, significantly improving the reliability of the generated code, reducing the large amount of manual debugging work caused by code errors, and achieving a higher degree of automation. This embodiment provides an automatic robot code generation method with environmental perception and self-correction capabilities. Based solely on high-level natural language instructions provided by the user, it can automatically complete the entire process from environmental perception to generating correct low-level control code, reducing the complexity of robot programming and improving the intelligence level of human-computer interaction.

[0032] In step S101 of some embodiments, the natural language task instruction is input by the user to the robot as the basis for the robot to perform subsequent actions. For example, the natural language task instruction may be "the robot moves to the apple on the table" or "grab the apple on the table".

[0033] In this embodiment, please refer to Figure 2. The robot motion control code generation system is based on a large language model agent and adopts a two-layer agent collaboration framework, including a first agent (controller agent) and a second agent (code generation agent), which decomposes the complex "language to code" task into two independent sub-tasks.

[0034] The first agent is the controller agent. Its functions include task understanding and environmental awareness. This agent receives natural language task instructions from the user. It interacts with the physical environment by generating and executing a series of scripts that invoke perception tools. The agent's final output is not robot control code, but a natural language plan containing precise target poses for the robot's actions. .

[0035] The second agent is KRL (KUKA Robot Language) agent. Its function is code generation and syntax correction. This agent receives natural language planning data from the controller agent. Its task is to translate it into syntactically correct code that conforms to the specifications of a specific robot language (such as KRL). .

[0036] In step S102 of some embodiments, referring to Figures 2 and 3, the controller agent... By calling a series of preset toolsets The process involves performing perception planning, generating and executing a perception planning script to obtain the geometric information of the target object in the world coordinate system, and then generating a natural language plan to describe the robot's final action based on the acquired geometric information. Image acquisition tools for acquiring scene images, Image segmentation tools for segmenting objects in an image, For visual retrieval tools used to identify targets based on text descriptions, This is a coordinate mapping tool used to convert the coordinates of an object in the image coordinate system to coordinates in the world coordinate system.

[0037] In some embodiments, step S102 may include, but is not limited to, steps S201 to S203: step S201, recognizing the natural language task instruction and determining the target object; step S202, calculating the coordinates of the target object in the perceived information to obtain the world coordinates of the target object; step S203, generating a natural language plan based on the natural language task instruction and the world coordinates.

[0038] In step S201 of some embodiments, according to the natural language task instructions Extracting the text description of the target object from the text description. (e.g., "apple").

[0039] In step S202 of some embodiments, the target object mentioned in natural language is located in its precise three-dimensional coordinates in the robot world coordinate system through visual recognition and coordinate transformation. , which refers to the world coordinates of the target object.

[0040] In some embodiments, step S202 may include, but is not limited to, steps S221 to S224: Step S221, performing image segmentation on the scene image to obtain the contour masks of each object; Step S222, calculating the similarity of the image regions corresponding to each contour mask according to the text description of the target object to determine the target mask of the target object; Step S223, calculating the geometric center of the target mask to obtain the pixel coordinates of the target object; Step S224, performing coordinate transformation on the pixel coordinates to obtain world coordinates.

[0041] In step S221 of some embodiments, an image segmentation tool is used. (e.g., SAM model) for RGB images All objects are segmented into instances to obtain a set of contour masks for each object. .

[0042] In step S222 of some embodiments, a visual retrieval tool is used. (Such as the CLIP model) identifies the target object, that is, it calculates each With each mask Corresponding image region The similarity between them is used to identify the target mask of the best matching target object. The specific formula is as follows: .

[0043] In step S223 of some embodiments, the target mask is... The geometric center is calculated to obtain its pixel coordinates in the two-dimensional image coordinate system. .

[0044] In step S224 of some embodiments, the coordinate mapping tool is invoked. , pixel coordinates Convert to the robot's three-dimensional coordinates in the world coordinate system This process utilizes a pinhole camera model and incorporates a pre-calibrated camera intrinsic parameter matrix. and extrinsic parameter matrix (Transformation matrix from camera to robot base). Specifically, based on pixel coordinates. and depth images Depth value found Then, the pixel coordinates are converted into 3D points in the camera coordinate system. The specific formula is as follows: After sorting, we get: ;in, For the camera intrinsic parameter matrix, for The homogeneous transformation matrix.

[0045] Finally, using the extrinsic parameter matrix camera coordinates Transform to world coordinates As shown in the following formula: .

[0046] In step S203 of some embodiments, a natural language plan is generated based on the obtained world coordinates. For example, the natural language plan may be "using PTP motion instructions to move the center point of the robot tool to the world coordinate position {X, Y, Z}".

[0047] The first agent (controller agent) actively acquires the precise geometric information of the task target in the robot coordinate system by calling the vision model and coordinate transformation tools. It transforms the vague natural language instructions into intermediate planning instructions containing clear three-dimensional coordinates, providing a reliable physical world basis for subsequent code generation.

[0048] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302: step S301, obtaining grammatical knowledge from a preset robot control language grammar knowledge base; step S302, generating code based on the grammatical knowledge and natural language planning to obtain initial robot motion control code.

[0049] In steps S301 to S302 of some embodiments, referring to Figures 2 and 4, the second agent uses a pre-set robot control language syntax knowledge base written by human experts as contextual enhancement information during the generation of robot motion control code, so as to improve the accuracy and standardization of the generated code.

[0050] In some embodiments, step S104 may include, but is not limited to, step S401: Step S401, performing syntax verification on the initial robot motion control code and generating a first feedback result, the first feedback result being used to iteratively correct the initial robot motion control code.

[0051] In step S401 of some embodiments, please refer to Figures 2 and 4, KRL agent After the code is generated, the system immediately invokes a KRL parser built on ANTLR4. The code is validated. The syntax parser performs syntax validation on the generated robot motion control code and provides error feedback to the second agent when syntax errors are detected, so that it can iteratively correct them.

[0052] In step S105 of some embodiments, the code that is corrected in each iteration is verified by a syntax parser until no code error is detected, or the number of iterations is greater than or equal to the preset number of iterations.

[0053] In some embodiments, step S105 may include, but is not limited to, steps S501 to S504: Step S501, correcting the initial robot motion control code according to the code error type and code error location in the first feedback result to obtain intermediate corrected code; Step S502, verifying the intermediate corrected code to obtain a second feedback result; Step S503, if the second feedback result is empty, using the intermediate corrected code as the target robot motion control code; Step S504, if the second feedback result is not empty, correcting the initial robot motion control code according to the code error type and code error location in the second feedback result.

[0054] In some embodiments, steps S501 to S504, as shown in Figure 4, constitute an iterative correction loop, whereby... Let be the number of iterations, initially. .

[0055] Step 1: Code generation. KRL proxy Receive the plan generated by the controller agent And the error feedback from the previous round (Initial time) (empty), generate the first Version KRL code This refers to the intermediate correction code, specifically as shown in the following formula: .

[0056] Step 2: Syntax validation. The syntax parser is invoked to validate the syntax. Perform the verification and obtain the feedback result. This refers to either the first or second feedback result: Feedback results It is a structured object; if the code syntax is correct, then... If errors exist, then Includes the error type, row number, column number, and error message.

[0057] Step 3: Loop through the conditional statements. If... If the code syntax is correct, the loop terminates and outputs the correct value. As the final result (target robot motion control code). If and ( (where the maximum number of attempts is preset), then let Return to step one and report the current error. Used for the next round of code generation. If If the loop terminates, the report generation fails.

[0058] This closed-loop mechanism of "generation-verification-feedback-correction" improves the high reliability of the code at the syntax level.

[0059] The closed-loop control method, which combines a second agent (code generation agent) with a syntax parser, performs real-time verification of the generated code and uses syntax errors as feedback signals to drive the second agent to perform iterative corrections, thereby improving the syntax correctness and executability of the final output code.

[0060] The following is a detailed description and explanation of the solution of this invention embodiment, using specific application examples: A. Taking the KUKA robot equipped with an Intel RealSense depth camera as an example, hand-eye calibration is performed. Before the system is officially run, a calibration process with the eye outside the hand needs to be executed. By moving the calibration board marked with Aruco within the robot's workspace and simultaneously acquiring the robot's end-effector pose and camera images, the extrinsic parameter matrix E of the camera relative to the robot base, and the intrinsic parameter matrix K of the camera itself are calculated and saved for subsequent coordinate transformation.

[0061] B. User Input Natural Language Task For example: "Move the robot next to the apple on the table." Controller agent Accept this task.

[0062] C. Controller Agent The perception planning process begins. It first generates instructions to invoke the image acquisition tool to obtain RGB images of the scene. and depth images Next, instructions are generated to invoke image segmentation and visual retrieval tools, locate the "apple" in the image, and obtain its pixel coordinates. Finally, generate the command to call the coordinate mapping tool to convert the apple's pixel coordinates to world coordinates, for example, to get the result (X=1120.3,Y=43.7,Z=458.0).

[0063] D. Controller Agent Based on the above perception results, a natural language planning function is generated. "Use the PTP motion command to move the robot tool center point to the world coordinate position (1120.3,43.7,458.0)."

[0064] E, KRL Agent Receive the plan and generate the first version of KRL code. This code may contain syntax flaws, such as omitting the attitude and configuration fields S and T when defining the E6POS variable.

[0065] F. System call to KRL syntax parser For the first version of the code During validation, the parser detected a syntax error and returned an error response. "The E6POS structure definition is incomplete, missing S and T fields."

[0066] G, KRL Agent This error feedback has been received. and in conjunction with the original plan It will perform a second generation, this time filling in missing fields and generating a syntactically correct second version of KRL code. .

[0067] H, Second Edition Code After being validated again by the syntax parser, this validation passed and returned... The system will use this syntactically correct KRL code. As the final output, this code can be directly used by the robot controller for execution.

[0068] The embodiments of this application have at least the following beneficial effects: First, by introducing a controller agent with visual perception capabilities, the barrier between the language model and the physical world is broken down, so that the generated robot motion trajectory can accurately correspond to objects in the real world, ensuring the accuracy of task execution.

[0069] Secondly, the closed-loop correction mechanism, through real-time syntax verification and error feedback, endows the code generation process with the ability to self-correct, significantly improving the reliability of the generated code, reducing a large amount of manual debugging work caused by syntax errors, and achieving a higher degree of automation.

[0070] By employing a two-layer agent architecture, complex tasks are broken down into two stages: a high-level perception and planning stage and a low-level code generation stage. The first agent is responsible for interacting with the physical world, addressing the questions of "what to do" and "where to go," while the second agent focuses on generating syntactically correct code, addressing the question of "how to write it." Utilizing the closed-loop feedback mechanism of the syntax parser significantly improves the reliability of code generation, achieving automated and high-precision generation from natural language to executable robot code.

[0071] This application also provides a robot motion control code generation system that can implement the above-described method. The system includes: an acquisition module for acquiring natural language task instructions; a parsing module for parsing the robot's perception information according to the natural language task instructions to obtain a natural language plan; a generation module for generating code according to the natural language plan to obtain initial robot motion control code; a verification module for verifying the initial robot motion control code to determine whether there are any errors in the initial robot motion control code; and a correction module for iteratively correcting the initial robot motion control code if there are any errors to obtain the target robot motion control code.

[0072] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0073] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0074] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] Please refer to Figure 5, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 901, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 902, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901. The input / output interface 903 is used to implement information input and output. The communication interface 904 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (such as USB, network cable, etc.) or by wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 905 transmits information between the various components of the device (such as the processor 901, memory 902, input / output interface 903 and communication interface 904). The processor 901, memory 902, input / output interface 903 and communication interface 904 realize communication connection between each other within the device through the bus 905.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0077] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0078] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0079] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0080] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0081] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0082] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0083] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0086] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0087] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating robot motion control code, characterized in that, The method includes the following steps: acquiring natural language task instructions; parsing the robot's perception information according to the natural language task instructions to obtain natural language planning; generating code according to the natural language planning to obtain initial robot motion control code; verifying the initial robot motion control code to determine whether there are errors in the initial robot motion control code; if there are errors in the initial robot motion control code, iteratively correcting the initial robot motion control code to obtain target robot motion control code.

2. The method according to claim 1, characterized in that, The step of parsing the robot's perception information based on the natural language task instructions to obtain natural language planning includes: recognizing the natural language task instructions to determine the target object; calculating the coordinates of the target object in the perception information to obtain the world coordinates of the target object; and generating the natural language planning based on the natural language task instructions and the world coordinates.

3. The method according to claim 2, characterized in that, The perceived information includes a scene image. The step of calculating the coordinates of the target object in the perceived information to obtain the world coordinates of the target object includes: performing image segmentation on the scene image to obtain contour masks for each object; calculating the similarity of the image region corresponding to each contour mask based on the text description of the target object to determine the target mask of the target object; calculating the geometric center of the target mask to obtain the pixel coordinates of the target object; and performing coordinate transformation on the pixel coordinates to obtain the world coordinates.

4. The method according to claim 1, characterized in that, The step of generating initial robot motion control code based on the natural language planning includes: acquiring grammatical knowledge from a preset robot control language grammar knowledge base; and generating code based on the grammatical knowledge and the natural language planning to obtain initial robot motion control code.

5. The method according to claim 1, characterized in that, The step of verifying the initial robot motion control code includes: performing syntax verification on the initial robot motion control code and generating a first feedback result, wherein the first feedback result is used to iteratively correct the initial robot motion control code.

6. The method according to claim 5, characterized in that, If the initial robot motion control code contains errors, iterative correction is performed on the initial robot motion control code to obtain the target robot motion control code. This includes: correcting the initial robot motion control code according to the code error type and code error location in the first feedback result to obtain intermediate corrected code; verifying the intermediate corrected code to obtain a second feedback result; if the second feedback result is empty, the intermediate corrected code is used as the target robot motion control code; if the second feedback result is not empty, the initial robot motion control code is corrected according to the code error type and code error location in the second feedback result.

7. A robot motion control code generation system, characterized in that, The system includes: an acquisition module for acquiring natural language task instructions; a parsing module for parsing the robot's perception information according to the natural language task instructions to obtain a natural language plan; a generation module for generating code according to the natural language plan to obtain initial robot motion control code; a verification module for verifying the initial robot motion control code to determine whether there are any errors in the initial robot motion control code; and a correction module for iteratively correcting the initial robot motion control code if there are any errors to obtain target robot motion control code.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; when said at least one program is executed by said at least one processor, such that said at least one processor implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images