Robot modular simulation system and method based on LLM and electronic equipment

Abstract

The invention relates to the technical field of robot simulation, in particular to an LLM-based robot modular simulation system and method and electronic equipment. The system comprises an LLM module, a visual algorithm module and a control module. Compared with the prior art, the robot simulation process based on the LLM is provided, and the LLM model is introduced into the robot simulation process. Besides, all functions in a traditional robot simulation process are decoupled, function packaging is carried out in a modularized mode, all modules communicate in an ROS Topic mode, and therefore the problems that the functions are integrated in the current robot simulation debugging process, and the debugging flexibility is poor are solved.

Description

Technical Field

[0001] This application relates to the field of robot simulation technology, and in particular to a modular robot simulation system, method and electronic device based on LLM. Background Technology

[0002] With the development of artificial intelligence and robotics, robot systems are becoming increasingly complex. Robots are increasingly reliant on simulation environments for rapid development and iteration. Furthermore, the development of LLM (Large Language Model) has added simulation requirements for natural language interaction and task understanding and execution, in addition to the conventional simulation and debugging of vision and control algorithms.

[0003] Currently, robot simulation and debugging typically involves building an integrated simulation platform using various programming languages ​​based on simulation software. Functionality is then modified by altering the code to debug the robot's capabilities. This approach requires complete code re-editing for each debugging session, making it inconvenient and inflexible for adjusting individual module functionalities.

[0004] Therefore, there is an urgent need for a more flexible robot debugging and simulation process. Summary of the Invention

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides a modular robot simulation system, method and electronic device based on LLM, the main purpose of which is to solve the problem of poor debugging flexibility in the current robot simulation debugging process due to its integrated functions.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a modular robot simulation system based on LLM, comprising: The LLM module is used to respond to user-issued natural language commands based on the simulation environment, parse the natural language commands into structured task data, and determine the execution module and execution order to which the task data belongs. The vision algorithm module is used to receive task data distributed by the LLM module, perform image processing based on the preset callable tools and processing algorithms configured by the robot, and publish the image processing results in real time through ROS Topic. The control module is used to receive task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generate sub-task driving instructions according to the robot's preset drivable robot hardware nodes, and publish them to the designated hardware nodes in real time through ROS Topic; and obtain the execution results of each driving instruction fed back to the LLM module. The LLM module is also used to receive image processing results published by the vision algorithm module and / or execution results of the driving instructions fed back by the control module, and update the structured task data and execution order to complete the natural language instructions.

[0007] Optionally, the LLM-based modular robot simulation system further includes a robot asset module for providing predefined asset information used in the simulation environment. The predefined asset information includes preset callable tools and processing algorithms configured for the robot, preset driveable robot hardware nodes, and simulation environment materials. In addition, the robot asset module interacts with the LLM module, the control module, and the vision algorithm module through the ROS Topic publish-subscribe mechanism.

[0008] Optionally, in the LLM-based modular robot simulation system, the steps for creating simulation environment materials in the robot asset module include: acquiring multi-view images of the target object using multiple cameras to obtain point cloud data of the target object; obtaining a model file of the target object based on the camera intrinsics of the multiple cameras and the point cloud data of the target object; importing the model file into the Isaac Sim simulation environment, and in response to user input, adding physical attributes and ROS Topic communication interfaces to the model file and packaging it to complete the material creation of the target object.

[0009] Optionally, after obtaining the point cloud data of the target object, the creation step further includes: determining whether the point cloud data of the target object is complete on each surface; if the point cloud data is incomplete, marking the surface of the current acquisition surface with an ArUco QR code and flipping the target object; and acquiring the point cloud data of the target object on each surface based on the posture data of the ArUco QR code, until complete point cloud data of the target object on each surface is acquired.

[0010] Optionally, the LLM-based modular robot simulation system also includes an interaction module for modifying the configuration of the robot simulation environment in response to user operation commands.

[0011] Optionally, modifying the configuration of the robot simulation environment includes: loading or unloading the LLM module, the control module, or the vision algorithm module during the simulation process; modifying the asset information of the robot asset module; configuring the operating parameters of the LLM module, the control module, or the vision algorithm module in real time; and responding to the user's manual control command by calling and executing the recognition function of the vision algorithm module, or calling the control module to control the robot to complete a specified action.

[0012] Optionally, the asset information of the robot asset module can be configured and modified, including: updating the robot's internal preset callable tools and processing algorithms, or updating the simulation environment materials, or updating the robot's preset driveable robot hardware nodes.

[0013] Optionally, the interaction module is inserted via an Isaac Sim plugin and has a built-in ROS communication interface.

[0014] Secondly, embodiments of this application provide an LLM-based modular robot simulation method, applied to an LLM-based modular robot simulation system as described in any one of the first aspects above, the method comprising: The LLM module receives natural language commands issued by the user in response to the simulation environment, parses the natural language commands into structured task data, and determines the execution module and execution order to which the task data belongs. The vision algorithm module receives task data distributed by the LLM module and performs image processing based on the preset callable tools and processing algorithms configured by the robot, and publishes the image processing results in real time through ROS Topic. The control module receives task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generates sub-task driving instructions based on the robot's preset drivable robot hardware nodes, and publishes them to the designated hardware nodes in real time through ROS Topic; and obtains the execution results of each driving instruction fed back to the LLM module. The LLM module receives image processing results from the vision algorithm module and / or execution results of drive instructions from the control module, updates the structured task data and execution order, and completes the natural language instructions.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the LLM-based modular robot simulation method described in the second aspect.

[0016] Based on the above technical solutions, this application provides a modular robot simulation system, method, and electronic device based on LLM. The system includes an LLM module, used to respond to natural language commands issued by the user based on the simulation environment, parse the natural language commands into structured task data, and determine the execution module and execution order to which the task data belongs; a vision algorithm module, used to receive task data distributed by the LLM module and perform image processing based on preset callable tools and processing algorithms configured by the robot, and publish the image processing results in real time through ROS Topic; a control module, used to receive task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generate sub-task driving instructions according to the robot's preset driveable robot hardware nodes, and publish them to the specified hardware nodes in real time through ROS Topic; and obtain the execution results of each driving instruction fed back to the LLM module; the LLM module is also used to receive image processing results published by the vision algorithm module and / or execution results of the driving instructions fed back by the control module, update the structured task data and execution order to complete the natural language commands. Compared with related technologies, this application proposes an LLM-based robot simulation process, which decouples the various functions in the traditional robot simulation process and encapsulates the functions in a modular way. The modules communicate with each other through ROS Topic, thereby solving the problem of poor debugging flexibility in the current robot simulation debugging process due to its integrated functions. Attached Figure Description

[0017] Figure 1 A schematic diagram of a modular robot simulation system based on LLM provided for embodiments of this application; Figure 2 A flowchart illustrating the asset creation steps in a robot asset module provided in this application embodiment; Figure 3 A schematic diagram of another LLM-based modular robot simulation system provided in this application embodiment. Detailed Implementation

[0018] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0019] As mentioned in the background, with the development of artificial intelligence and robotics, robot systems are becoming increasingly complex. The fields involved, such as mechanics, electrical engineering, and algorithms, are expanding, and robots are increasingly reliant on simulation environments for rapid development and iteration—the so-called robot simulation. Furthermore, with the development of LLM (Large Language Model), in addition to the conventional simulation and debugging needs for vision and control algorithms, robots now also require simulation for natural language interaction and task understanding and execution.

[0020] Currently, staff typically use various programming languages ​​to build integrated simulation platforms based on simulation software, and then modify the code to adjust functionality and debug the robot's capabilities. However, this approach is neither convenient for interaction and debugging, and often requires extensive overall code modifications after changes in requirements, resulting in poor flexibility in the debugging process.

[0021] To address this issue, this application proposes an LLM-based modular robot simulation system. This system can be combined with... Figure 1 As shown, the system includes: The LLM module is used to respond to user-issued natural language commands based on the simulation environment, parse the natural language commands into structured task data, and determine the execution module to which the task data belongs and the execution order. The vision algorithm module is used to receive task data distributed by the LLM module, perform image processing based on the preset callable tools and processing algorithms configured by the robot, and publish the image processing results in real time through ROS Topic. The control module is used to receive task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generate sub-task driving instructions according to the robot's preset driveable robot hardware nodes, and publish them to the specified hardware nodes in real time through ROS Topic; and obtain the execution results of each driving instruction fed back to the LLM module. The LLM module is also used to receive image processing results from the vision algorithm module and / or execution results of drive instructions from the control module, update structured task data and execution order to complete natural language instructions.

[0022] In the robot simulation system of this embodiment, firstly, each function adopts a modular design. Each module only needs to encapsulate the relevant code to implement its own function, thereby decoupling the integrated simulation method in related technologies and providing a basis for subsequent flexible simulation adjustments.

[0023] Specifically, in the LLM module, which is also known as the Large Language Model module, existing large language models, such as ChatGpt, can be loaded to analyze and respond to user natural language. These models respond to user-issued natural language commands based on the simulation environment, parse the natural language commands into structured task data, and determine the execution module and execution order to which the task data belongs.

[0024] First, the natural language commands issued by the user based on the simulation environment refer to the instructions given by the user to the simulation process, such as "lift the left arm of the robotic arm" or "bring me the water glass on the table." Then, the natural language commands are parsed into structured task data, and the execution module and execution order of the task data are determined. Specifically, the structured task data includes task parameters for different tasks, task types to distinguish different tasks, task command numbers for ordering task execution, and relevant asset and resource information that may be used. The large language model determines the task execution module and the order of task execution based on this content of the structured task data.

[0025] Taking the two examples above, both are instructions given by the user to the simulation process, but different execution methods will be used in the subsequent execution. "Raise the left arm of the robotic arm" is a natural language instruction that does not involve interaction with the environment. Therefore, in the structured task data parsed from it, the task type is direct control (direct control does not require interaction with the environment and can be preset), and the task parameter is 1 (meaning the control parameter corresponding to raising the left arm of the robotic arm). Based on the task type and task parameters, it can be determined that the execution module can be either the LLM module directly controlling the robot's movement, or the control module controlling the robot's movement. "Bring me the water glass on the table" is a natural language command involving interaction with the environment. Therefore, the structured task data obtained from its parsing is classified as an environment-interaction type (environment-interaction type involves interacting with the environment and can be preset). The task parameters are 2, 3, and 4 (2 means image recognition through the vision algorithm module, 3 means grasping the glass by controlling the robot arm based on the image recognition results, and 4 means moving the glass to the user; this meaning is only an example and does not limit the content of the task parameters). The command number is used to limit the order of multiple sub-tasks, for example, limiting 2 before 3, 3 before 4, and the asset resource information is the glass material.

[0026] In the vision algorithm module, task data distributed by the LLM module is received and image processing is performed based on the preset callable tools and processing algorithms configured for the robot. The preset callable tools configured for the robot are a type of robot asset; for example, the camera or image acquisition sensor configured on the robot itself is a type of robot asset. The corresponding acquisition and processing methods of the camera or image acquisition sensor are the corresponding processing algorithms. Finally, the image processing results are published in real time via ROS Topic.

[0027] In the control module, based on different task types, it receives task data distributed by the LLM module and / or image processing results published by the vision algorithm module, and then generates sub-task driving instructions according to the robot's preset drivable robot hardware nodes. These preset drivable robot hardware nodes refer to the robot's own components, such as the robot arm. The control module then publishes sub-task driving instructions (e.g., raising the arm) to the designated hardware nodes in real time via ROS Topic. Furthermore, the control module also obtains the execution results of the driving instructions fed back to the LLM module. The control module also needs to return the execution results to the LLM module in the form of a service request to notify the decision center of the robot's movement status, thereby enabling better control of the robot's next movement.

[0028] The LLM module also obtains image processing results from the vision algorithm module and / or execution results from the control module in a service and request manner to adjust the subsequent motion mode, thereby dealing with different simulation debugging scenarios.

[0029] It's important to note that the LLM module doesn't use ROS Topic communication for task distribution. Instead, it directly sends tasks to the control or vision algorithm module for rapid execution, avoiding communication delays and ensuring accurate task distribution. However, it obtains results from the control and vision algorithm modules through ROS Topic subscription, as multiple modules may need these results simultaneously. For example, if a natural language instruction is broken down into multiple tasks, and the control module needs to process the results from the vision algorithm module, while the LLM model also needs to provide feedback to the user, then in this scenario, the LLM and control modules subscribe to content published by the vision algorithm module within the same ROS Topic to execute subsequent tasks.

[0030] In this embodiment, the LLM module first responds to user-issued natural language commands based on the simulation environment, parses the natural language commands into structured task data, and determines the execution module and execution order to which the task data belongs. The vision algorithm module receives the task data distributed by the LLM module and performs image processing based on preset callable tools and processing algorithms configured for the robot, and publishes the image processing results in real time through ROS Topic. The control module receives the task data distributed by the LLM module and / or the image processing results published by the vision algorithm module, generates sub-task driving instructions according to the robot's preset driveable robot hardware nodes, and publishes them to the designated hardware nodes in real time through ROS Topic; and obtains the execution results of the driving instructions fed back to the LLM module. The LLM module also receives the image processing results published by the vision algorithm module and / or the execution results of the driving instructions fed back by the control module, and updates the structured task data and execution order.

[0031] Compared with related technologies, this embodiment proposes an LLM-based robot simulation process, which decouples the various functions in the traditional robot simulation process and encapsulates the functions in a modular way. The modules communicate with each other through ROS Topic, thereby solving the problem of poor debugging flexibility in the current robot simulation debugging process due to its integrated functions.

[0032] Optionally, the robot asset module provides predefined asset information used in the simulation environment. The predefined asset information includes preset callable tools and processing algorithms for robot configuration, preset driveable robot hardware nodes, and simulation environment materials. In addition, the robot asset module interacts with the LLM module, control module, and vision algorithm module through the ROS Topic publish and subscribe mechanism.

[0033] In this embodiment, the robot modular simulation system also includes a robot asset module, which can be combined with... Figure 1As shown, the robot asset module provides predefined asset information used in the simulation environment. As briefly described in the above scheme, the predefined asset information used in the simulation environment specifically includes preset callable tools and processing algorithms configured for the robot, preset driveable robot hardware nodes, and simulation environment materials. The preset callable tools configured for the robot include tools configured within the robot itself, such as cameras; the processing algorithms are the processing algorithms corresponding to the tools. The preset driveable robot hardware nodes include the robot's own components, such as robotic arms. Simulation environment materials refer to items that may be interactive in the simulation environment, such as chairs, tables, and cups. The robot asset module interacts with other modules through the ROS Topic publish-subscribe mechanism. By packaging the robot assets, on the one hand, it facilitates the simulation debugging process; the robot asset module only needs to subscribe to the drive instructions published by the control module to complete the corresponding control, without needing to modify the entire code during simulation as in related technologies. On the other hand, after packaging the asset module, the assets and their communication functions can be packaged together and ported to the next simulation environment, allowing for direct use after porting, thus facilitating asset portability.

[0034] Optionally, the steps for creating simulation environment materials in the robot asset module include: acquiring multi-view images of the target object using multiple cameras to obtain point cloud data of the target object; obtaining the model file of the target object based on the camera intrinsics of the multiple cameras and the point cloud data of the target object; importing the model file into the Isaac Sim simulation environment, and in response to user input, adding physical attributes and ROS Topic communication interfaces to the model file and packaging it to complete the material creation of the target object.

[0035] Furthermore, after obtaining the point cloud data of the target object, the creation steps also include: determining whether the point cloud data of the target object is complete on each surface; if the point cloud data is incomplete, marking the surface of the current acquisition surface with an ArUco QR code and flipping the target object; and acquiring the point cloud data of the target object on each surface based on the posture data of the ArUco QR code, until complete point cloud data of the target object on each surface is acquired.

[0036] In this embodiment, materials in the simulation environment can be added, and the adding steps can be combined with... Figure 2As shown, firstly, multiple cameras mounted on the robot at different perspectives acquire color and depth images of the target object. Then, it is determined whether the point cloud data of the target object is complete on each surface. This determination is made by colorizing the point cloud using the intrinsic parameters of each camera and then judging whether the object is complete. If incomplete, an ArUco QR code (a marker used for localization) is placed on the current acquisition surface of the target object, and the object is flipped. During the flipping process, based on the pose data of the ArUco QR code, point cloud data of the target object on each surface is acquired until complete point cloud data of the target object on each surface is acquired and then stitched together to obtain the complete point cloud data of the target object.

[0037] The complete point cloud data of the target object is then triangulated (also known as surface reconstruction) and imported into the mesh model to obtain a model file of the target object in ".obj" format. This model file is then imported into Isaac Sim, where physical properties and a ROS Topic communication interface are added, and the file is packaged to complete the material creation of the target object. The final material creation is a ".USD" file. Files obtained in this way can be easily imported into other simulation environments and come with a built-in communication protocol; only the data to be sent to the model needs to be specified. This facilitates asset portability.

[0038] Optionally, the LLM-based modular robot simulation system also includes an interaction module for modifying the configuration of the robot simulation environment in response to user operation commands.

[0039] In this embodiment, the modular robot simulation system also includes an interaction module. This module responds to user commands and modifies the configuration of the robot simulation environment. Specifically, the interaction module includes a camera configuration tool for modifying camera configurations; a resource destruction tool for deleting and destroying existing resources; an environment generation and modification tool for setting and modifying the robot's simulation environment; a module hot-swap tool for replacing loaded modules during simulation debugging; and an algorithm invocation tool for calling corresponding execution algorithms based on different task requirements. As a feasible implementation, the interaction module provides users with an interactive graphical interface, allowing users to modify the robot simulation environment configuration through touch controls.

[0040] Furthermore, the configuration of the robot simulation environment is modified, including: loading or unloading the LLM module, control module, or vision algorithm module during the simulation process; configuring and modifying the asset information of the robot asset module; configuring the operating parameters of the LLM module, control module, or vision algorithm module in real time; responding to the user's manual control commands, calling and executing the recognition function of the vision algorithm module, or calling the control module to control the robot to complete the specified action.

[0041] This embodiment describes the modifiable configurations. Specifically, this may include changing the simulation algorithm, such as real-time configuration of the operating parameters of the LLM module, control module, or vision algorithm module. These operating parameters may include the recognition confidence threshold and camera parameters of the vision algorithm module, or the motion control parameters of the control module, etc. Interactive debugging is also included, such as directly calling and executing the recognition function of the vision algorithm module, or calling the control module to control the robot to complete a specified action. It also includes loading or unloading the LLM module, control module, or vision algorithm module, such as replacing the LLM model with other modules capable of understanding natural language. Furthermore, the robot asset information can be configured, modified, and adjusted. This interactive simulation method facilitates the simulation debugging process and improves simulation efficiency.

[0042] Furthermore, the asset information of the robot asset module can be configured and modified, including: updating the robot's internal preset callable tools and processing algorithms, or updating the simulation environment materials, or updating the robot's preset driveable robot hardware nodes.

[0043] In this embodiment, as Figure 1 As shown, the robot asset module specifically includes pre-set, callable tools and processing algorithms within the robot, simulation environment materials, and pre-set, driveable robot hardware nodes. Their specific meanings have already been explained above and will not be repeated here. Users can modify this asset information through the interactive module, such as deleting a robotic arm. This enables rapid configuration adjustments and improves simulation efficiency.

[0044] Optionally, the interaction module is inserted via an Isaac Sim plugin and has a built-in ROS communication interface.

[0045] In this embodiment, the interaction module is designed based on the Isaac Sim plugin and can be packaged as a separate file for import into other simulation systems, achieving ease of portability. Furthermore, the built-in ROS communication interface can also be directly used in the ported system, thereby greatly improving simulation efficiency.

[0046] The LLM-based modular robot simulation system provided by any of the above embodiments has at least the following technical effects: (1) A standard LLM intelligent robot simulation process was established, which made up for the shortcomings of the current integrated intelligent simulation platform, and a standardized simulation process paradigm was proposed.

[0047] (2) Modularize the various functions of the LLM intelligent robot simulation process so that it can be easily modified and replaced in various aspects such as changing the simulation algorithm, packaging the assets required for simulation, replacing the LLM model, and interaction design, and can be combined with each other in multiple workflows.

[0048] (3) By introducing an interactive module, an interactive simulation environment is provided, which facilitates simulation debugging and improves simulation efficiency. The Isaac Sim design of its plugins also realizes modular portability.

[0049] (4) It provides a lifecycle management system from physical modeling to simulation to destruction. With the help of Isaac Sim’s ability to package ROS Topic, assets and their communication functions can be packaged together as assets, which greatly facilitates the transfer of assets.

[0050] Based on the above embodiments, a modular robot simulation method based on LLM is also proposed, applicable to any of the LLM-based modular robot simulation systems described above; the method includes: The LLM module receives natural language commands issued by the user in response to the simulation environment, parses the natural language commands into structured task data, and determines the execution module and execution order to which the task data belongs. The vision algorithm module receives task data distributed by the LLM module and performs image processing based on the preset callable tools and processing algorithms configured by the robot, and publishes the image processing results in real time through ROS Topic. The control module receives task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generates sub-task driving instructions based on the robot's preset drivable robot hardware nodes, and publishes them to the designated hardware nodes in real time through ROS Topic; and obtains the execution results of each driving instruction fed back to the LLM module. The LLM module receives image processing results from the vision algorithm module and / or execution results of drive instructions from the control module, updates the structured task data and execution order, and completes the natural language instructions.

[0051] By using an LLM-based modular simulation method for robots, combined with Figure 3As shown, the simulation process proposed in the embodiments of this application is described in its entirety. First, the user or simulation engineer issues debugging commands, which may include natural language commands or direct debugging control by the user through the interactive module.

[0052] When users directly debug and control the system through the interactive module, they can manage the configuration and algorithms of the LLM module, control algorithm module, vision algorithm module, and robot asset module. Specifically, management can include loading or deleting entire modules, such as replacing the LLM module with another language model; it can also involve adjusting specific module parameters, such as configuring the operating parameters of the control module or vision algorithm module in real time. These operating parameters can include the recognition confidence threshold and camera parameters of the vision algorithm module, or the motion control parameters of the control module, etc.

[0053] When users debug via natural language commands, the LLM module first parses the commands to obtain structured task data. This data includes task type and parameters to determine the task execution module and execution order. The corresponding task is then sent to the control module (referred to as the control algorithm module in the diagram) and / or the vision algorithm module. It should be noted that the control module also subscribes to the processing results of the vision algorithm module in some tasks; this is only performed in certain cases and is not shown in the diagram. Finally, the robot asset module (referred to as the robot USD asset module in the diagram) completes the debugging process by subscribing to the commands of the control and vision algorithm modules. Specifically, the control module can also parse complex movements, such as moving the end effector of a robotic arm to a certain position. This requires the use of tools such as MoveIt or Pinochle to calculate the required rotation angles of each joint of the robotic arm. Furthermore, the control module needs to return the execution results to the LLM module via service requests to inform the decision center of the robot's movement status and prepare for the next movement.

[0054] Except for the LLM module, which directly sends natural language instructions to designated execution modules when parsing and distributing tasks, communication between modules is achieved through the publish / subscribe mechanism of ROS Topic. This is because directly distributing tasks to execution modules avoids communication delays, enables rapid execution, and ensures the accuracy of task distribution. Other implementable methods can be applied in accordance with the methods described in the above system embodiments, and will not be elaborated upon here.

[0055] To achieve the above objectives, this application also provides an electronic device that can be configured on a computer side, etc. The device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the methods proposed in the above embodiments.

[0056] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0057] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0058] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the solution of this embodiment, compared with related technologies, a robot simulation process based on LLM is proposed, which decouples the various functions in the traditional robot simulation process and encapsulates the functions in a modular way. The modules communicate with each other through ROS Topic, thereby solving the problem of poor debugging flexibility in the current robot simulation debugging process due to its integrated functions.

[0060] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0061] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0062] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A modular robot simulation system based on LLM, characterized in that, include: The LLM module is used to respond to user-issued natural language commands based on the simulation environment, parse the natural language commands into structured task data, and determine the execution module and execution order to which the task data belongs. The vision algorithm module is used to receive task data distributed by the LLM module, perform image processing based on the preset callable tools and processing algorithms configured by the robot, and publish the image processing results in real time through ROS Topic. The control module is used to receive task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generate sub-task driving instructions according to the robot's preset driveable robot hardware nodes, and publish them to the designated hardware nodes in real time through ROSTopic; and obtain the execution results of each driving instruction fed back to the LLM module. The LLM module is also used to receive image processing results published by the vision algorithm module and / or execution results of the driving instructions fed back by the control module, and update the structured task data and execution order to complete the natural language instructions.

2. The LLM-based modular robot simulation system according to claim 1, characterized in that, Also includes: The robot asset module is used to provide predefined asset information used in the simulation environment; the predefined asset information includes the preset callable tools and processing algorithms configured for the robot, the preset driveable robot hardware nodes, and simulation environment materials; Furthermore, the robot asset module interacts with the LLM module, the control module, and the vision algorithm module through the publish-subscribe mechanism of ROS Topic.

3. The LLM-based modular robot simulation system according to claim 2, characterized in that, The steps for creating simulation environment materials in the robot asset module include: The point cloud data of the target object is obtained by acquiring multi-view images of the target object using multiple cameras; Based on the camera intrinsic parameters of the multi-camera system and the point cloud data of the target object, the model file of the target object is obtained; The model file is imported into the Isaac Sim simulation environment, and in response to user input, physical properties and ROS Topic communication interfaces are added to the model file and packaged to complete the material creation of the target object.

4. The LLM-based modular robot simulation system according to claim 3, characterized in that, After obtaining the point cloud data of the target object, the creation step further includes: Determine whether the point cloud data of the target object is complete on each surface; In the case of incomplete point cloud data, mark the surface of the current acquisition surface with an ArUco QR code and flip the target object. Based on the posture data of the ArUco QR code, point cloud data of the target object on each face is collected until complete point cloud data of the target object on each face is collected.

5. The LLM-based modular robot simulation system according to claim 2, characterized in that, Also includes: The interaction module is used to modify the configuration of the robot simulation environment in response to user operation commands.

6. The LLM-based modular robot simulation system according to claim 5, characterized in that, The modification of the robot simulation environment configuration includes: During the simulation process, the LLM module, the control module, or the vision algorithm module may be loaded or unloaded. The asset information of the robot asset module is configured and modified; The operating parameters of the LLM module, the control module, or the vision algorithm module are configured in real time. In response to the user's manual control command, the vision algorithm module's recognition function is invoked and executed, or the control module is invoked to control the robot to complete the specified action.

7. The LLM-based modular robot simulation system according to claim 6, characterized in that, The configuration modification of the asset information in the robot asset module includes: Update the robot's internal preset callable tools and processing algorithms, or update the simulation environment materials, or update the robot's preset driveable robot hardware nodes.

8. The LLM-based modular robot simulation system according to claim 5, characterized in that, The interaction module is inserted via an Isaac Sim plugin and has a built-in ROS communication interface.

9. A modular simulation method for robots based on LLM, characterized in that, Applied to the LLM-based modular robot simulation system as described in any one of claims 1-8; the method includes: The LLM module receives natural language commands issued by the user in response to the simulation environment, parses the natural language commands into structured task data, and determines the execution module and execution order to which the task data belongs. The vision algorithm module receives task data distributed by the LLM module and performs image processing based on the preset callable tools and processing algorithms configured by the robot, and publishes the image processing results in real time through ROS Topic. The control module receives task data distributed by the LLM module and / or image processing results published by the vision algorithm module, generates sub-task driving instructions based on the robot's preset drivable robot hardware nodes, and publishes them to the designated hardware nodes in real time via ROSTopic; and obtains the execution results of each driving instruction fed back to the LLM module. The LLM module receives image processing results from the vision algorithm module and / or execution results of drive instructions from the control module, updates the structured task data and execution order, and completes the natural language instructions.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of claim 9.

Images