Multi-agent cooperation based embodied intelligence application software development system and method

Abstract

The application relates to the technical field of robot development, in particular to a somatic intelligent application software development system and method based on multi-agent cooperation, which comprises a user interaction subsystem, a multi-agent cooperation arrangement center connected with the user interaction subsystem and comprising a scene definition agent, an environment perception agent and a visualization agent, a simulation running engine connected with the multi-agent cooperation arrangement center, and an application construction and deployment module connected with the user interaction subsystem. The application significantly reduces the threshold of robot application development, realizes intent-driven visual development for non-professionals, solves the technical problems of lack of environment context in code generation by a large language model and easy occurrence of "space illusion", balances the efficiency of automatic generation and the interpretability of manual intervention, improves the development fault tolerance, greatly reduces the difficulty of debugging and error correction, realizes a full-process closed loop from "development" to "delivery", and shortens the product market cycle.

Description

Technical Field

[0001] This invention relates to the field of robot development technology, specifically to an embodied intelligent application software development system and method based on multi-agent collaboration. This invention is not only applicable to industrial robots and service robots, but can also be extended to other embodied intelligent systems that need to physically interact with the environment, such as autonomous vehicles and intelligent robotic arms. Background Technology

[0002] With the development of Industry 4.0 and embodied intelligence, the application scenarios of industrial robots and service robots are becoming increasingly complex. Traditional robot development models typically rely on professional integrated development environments (IDEs), such as ABB RobotStudio or various brands' self-developed IDEs. Developers need to have professional programming knowledge (such as C++, Python, or brand-specific scripting languages ​​like RAPID and KRL) to define the robot's actions, logic, and exception handling by writing code or dragging and dropping teach pendants.

[0003] The existing technology is as follows, along with its shortcomings: (1) Traditional teaching programming: The operator holds a teaching pendant, manually moves the robot to a designated point and records the location.

[0004] Disadvantages: Inefficient, unable to handle complex logic (such as branch judgment after visual recognition), and lacks versatility.

[0005] (2) Graphical / low-code programming (Blockly / Scratch class): Connect by dragging and dropping encapsulated function blocks.

[0006] Drawbacks: Although the grammatical threshold has been lowered, users still need to have rigorous logical thinking skills and manually configure detailed parameters for each node (such as coordinates, speed, and tool parameters), making it difficult to understand vague business intentions.

[0007] Existing AI-assisted programming (Copilot class): an LLM-based code completion tool.

[0008] Limitations: Most of these tools remain at the "text-to-code" level, and the generated code snippets often lack contextual association with the real hardware environment (physical simulation environment) (such as not knowing the specific coordinates of the shelf), and the generated code is difficult to directly perform full-process visualization verification and application encapsulation.

[0009] The common shortcoming of the aforementioned existing technologies is the lack of an end-to-end solution that goes directly from "natural language business intent" to "executable and verifiable robot applications," especially the lack of a mechanism for multi-agents to automatically extract environmental parameters in a simulation environment to complete the task logic. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention proposes an embodied intelligent application software development system and method based on multi-agent collaboration.

[0011] The technical problem to be solved by this invention is achieved by the following technical solution: The embodied intelligence application software development system based on multi-agent collaboration includes a user interaction subsystem, a multi-agent collaboration orchestration center, a simulation runtime engine, and an application building and deployment module, among which: The user interaction subsystem provides a natural language dialogue window and a visual flowchart editor. A multi-agent collaborative orchestration center, connected to the user interaction subsystem, includes: The scenario-defined intelligent agent is used to parse natural language instructions input through the user interaction subsystem and generate an abstract task sequence that includes the executing subject, action intention, and target object; The environment-aware intelligent agent is connected to the scene-defining intelligent agent and the simulation engine respectively. It is used to query the simulation engine to obtain the physical attribute data of the target object in the virtual environment based on the target object in the abstract task sequence, and map the physical attribute data to the abstract task sequence to generate a parameterized task sequence. The visual intelligent agent, connected to the user interaction subsystem and the environmental perception intelligent agent, is used to convert parameterized task sequences into visual task flowcharts and render and display them on the user interaction subsystem. The simulation engine is used to respond to query requests from environment-aware agents and provide physical parameters in the virtual environment. The application building and deployment module connects with the user interaction subsystem to convert visual task flowcharts into robot executable code and end-user applications.

[0012] As a further improvement of the present invention, the user interaction subsystem includes a multimodal input interface and a real-time feedback module connected to the multimodal input interface. The multimodal input interface integrates a chat window and a visual flowchart editor.

[0013] As a further improvement of the present invention, the simulation engine includes a scene graph database and a physical simulation kernel.

[0014] As a further improvement of the present invention, the application building and deployment module includes a parameter parser, an H5 generator, and a script compiler.

[0015] The method for developing embodied intelligence application software based on multi-agent collaboration, using the aforementioned embodied intelligence application software development system based on multi-agent collaboration, includes the following steps: Step S1, Intent Input and Intent Resolution: Users input natural language commands through the user interaction subsystem. The scene-defined intelligent agent parses the executing subject, action intent, and target object in the natural language commands to generate an abstract task sequence. Step S2, Environmental Parameter Fusion and Pose Calculation: The environmental perception agent queries the simulation engine based on the target object in the abstract task sequence to obtain the physical attribute data of the target object in the virtual scene. The environmental perception agent further calculates and generates the working pose and motion path of the robot's end effector relative to the target object based on the preset robot kinematics model and physical attribute data, and maps the physical attribute data to the abstract task sequence to generate a parameterized task sequence. Step S3: Visual flowchart generation and arrangement: The visual intelligent agent converts parameterized task sequences into visual task flowcharts containing logical connections, and renders them in the user interaction subsystem; Step S4, Interactive Incremental Update: In response to user modification commands to the visual task flowchart via natural language or graphical interaction, the environmentally aware agent identifies and updates only the environmental parameters of the task nodes associated with the modified content. The pre-computed paths associated with the updated nodes will be marked as 'invalid' and will be replanned during the final code compilation stage. The visual agent refreshes the visual task flowchart, updates the interface parameters, and marks the validity of the backend path planning cache in real time. Step S5: Application packaging and real device operation; The application build and deployment module is used for script compilation, application generation, and delivery.

[0016] As a further improvement of the present invention, the physical attribute data in step S2 includes at least: the unique identifier of the target object, three-dimensional spatial coordinates, three-dimensional dimensions and Euler angles.

[0017] As a further improvement of the present invention, the script compilation in step S5 specifically involves: compiling the task flowchart into robot control code and sending it to the target controller for execution.

[0018] As a further improvement of the present invention, the application generation in step S5 specifically involves: parsing the configurable parameter nodes in the task flowchart, and automatically generating a graphical user interface or H5 mobile application containing human-computer interaction controls based on the configurable parameter nodes.

[0019] The beneficial effects of this invention are: I. This invention significantly lowers the threshold for robot application development by using a natural language parsing mechanism based on scene-defined intelligent agents and the Vibe Coding interaction mode, enabling intention-driven visual development for non-professionals. Second, by introducing an environmentally perceptive intelligent agent and a simulation engine, this invention solves the technical problem of large language model code generation lacking environmental context and easily producing "spatial illusions". Third, this invention uses a visual intelligent agent to generate a visual task flowchart as an intermediate expression layer, instead of directly generating black box code. This balances the efficiency of automated generation with the interpretability of human intervention, improves the development fault tolerance rate, and greatly reduces the difficulty of debugging and error correction. Fourth, this invention automatically identifies configurable variables in the flowchart through the application building and deployment module and automatically encapsulates and generates H5 mobile applications and QR codes for delivery to end users, realizing a closed loop from "development" to "delivery" and shortening the product launch cycle. Fifth, this invention uses a script compiler in the application build and deployment module to compile the visualized task flowchart into the underlying code of different robots, so that the business logic generated by the same natural language can be easily deployed on hardware of different configurations or brands, thereby improving the system's compatibility and versatility with robots of different brands. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 System structure block diagram; Figure 2 This is a flowchart of the method. Figure 3 The interaction sequence diagram for the multi-agent collaborative orchestration center; Figure 4 Generate simulation renderings and animation diagrams for the user interface - dialogue interaction; Figure 5 Draw a flowchart for the user interface - dialogue interaction; Figure 6 A schematic diagram illustrating the automatic encapsulation of user interfaces from flowcharts to end-user applications.

[0021] In the diagram: 100, User Interaction Subsystem; 101, Multimodal Input Interface; 102, Real-time Feedback Module; 200, Multi-Agent Collaborative Orchestration Center; 201, Scene-Defining Agent; 202, Environment-Aware Agent; 203, Visual Agent; 300, Simulation Engine; 400, Application Building and Deployment Module. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0023] like Figure 1 As shown, the embodied intelligence application software development system based on multi-agent collaboration includes a user interaction subsystem 100, a multi-agent collaboration orchestration center 200, a simulation runtime engine 300, and an application building and deployment module 400. The various components communicate with each other via a data bus or API interface.

[0024] The user interaction subsystem 100 serves as the front end of the system. Its input end connects to the user to receive natural language commands and mouse operations, while its output end connects to the multi-agent collaborative orchestration center 200 and the application building and deployment module 400, respectively.

[0025] The user interaction subsystem 100 includes a multimodal input interface 101 and a real-time feedback module 102 connected to the multimodal input interface 101.

[0026] The multimodal input interface 101 integrates a chat window that supports natural language text input; it also includes a visual flowchart editor that allows users to drag, connect, and modify parameters of generated nodes.

[0027] The real-time feedback module 102 is used to render the flowchart generated by the backend in real time and display a preview window of the simulation screen.

[0028] The multi-agent collaborative orchestration center 200 is connected to the user interaction subsystem 100 and the simulation execution engine 300. It is used to convert fuzzy intentions into precise instructions. The multi-agent collaborative orchestration center 200 includes a scene definition agent 201, an environment perception agent 202, and a visualization agent 203.

[0029] The Scenario Definer 201 connects to the multimodal input interface 101 and is responsible for semantic understanding. It receives natural language commands from the user, extracts key elements such as the executing entity, the intent of the action, and the name of the target object, and outputs an abstract task architecture without physical parameters.

[0030] Compared to traditional robot development (such as using RobotStudio or ROS), which requires developers to have complex C++ / Python programming skills or knowledge of proprietary scripting languages ​​(such as RAPID) and to manually operate the teach pendant, this approach offers a more comprehensive solution.

[0031] This invention utilizes the natural language parsing mechanism of the scenario-defined intelligent agent 201 and the Vibe Coding interaction mode to allow users (such as supermarket managers or junior technicians) to drive the system to automatically generate logic simply through natural language (such as "develop a product display application"). This enables business personnel without a programming background to directly build industrial-grade or service-grade robot applications, greatly expanding the target audience for robot applications.

[0032] The environment-aware agent 202 (Context Grounding) connects with the scene-defining agent 201 and communicates bidirectionally with the simulation engine 300 via API. It queries the simulation engine for the physical attribute data of an object in virtual space based on the object name in the abstract task architecture. In this embodiment, this includes at least the target object's unique identifier, three-dimensional spatial coordinates, three-dimensional dimensions, and Euler angle pose. These physical parameters are then populated back into the task node, solving the technical problem of large language model-generated code lacking environmental context and easily producing "spatial illusions."

[0033] The environment-aware agent 202 initiates a query request to the simulation engine 300 via a RESTful API. The request body includes the name or label of the target object. The simulation engine 300 traverses the scene graph database and returns the unique identifier, pose matrix, bounding box size, and physical properties of the matching object. The environment-aware agent 202 is configured to automatically calculate the robot's end effector grasping pose and collision-free approach path based on the robot's kinematic model and the 3D spatial coordinates of the target object.

[0034] This invention introduces an environment-aware intelligent agent 202 to act as a "middleman," connecting natural language intent with the simulation engine's Scene Graph, automatically extracting real physical parameters (coordinates, size, ID) and filling them into the code logic; it solves the problem that existing LLM code generation (such as ChatGPT code writing) is often "illusion," as it does not know the specific (x, y, z) coordinates of the boxes in the scene, and the generated code cannot run.

[0035] The visualization agent 203 (Agent 3 - Flowchart Renderer) is connected to the environment-aware agent 202 and the real-time feedback module 102, and is responsible for visualization mapping. It maps the parameterized instruction set output by the environment-aware agent 202 into a standard flowchart data structure, such as JSON, and automatically lays out each functional node, such as start, recognition, movement, grasp, and end.

[0036] Compared to the "black box" mode of end-to-end large model control (VLA), where users cannot understand the internal logic or make fine adjustments if an error occurs, traditional graphical programming, while visual, requires dragging and dropping from scratch, resulting in low efficiency.

[0037] This invention uses a visual intelligent agent 203 to generate a visual task flowchart as an intermediate expression layer, rather than directly generating black-box code. Users can directly view logical branches on the flowchart and double-click nodes to modify parameters filled in by AI (such as the number of shelf layers). This approach achieves both instant generation of complex logic by AI (improving efficiency) and preservation of a structured flowchart for user verification and fine-tuning (ensuring controllability and interpretability). Users can modify robot behavior like editing a document, greatly reducing the difficulty of debugging and error correction.

[0038] The simulation runtime engine 300, connected to the multi-agent collaborative orchestration center 200, responds to query requests from the environment-aware agent 202 and provides physical parameters. The simulation runtime engine 300 includes a scene graph database and a physics simulation kernel. The scene graph database maintains real-time state data (position, hierarchy, physical attributes) of all objects (robots, shelves, workpieces) in the virtual environment. The physics simulation kernel provides collision detection, inverse kinematics (IK) algorithms, and path planning to ensure that the generated actions are physically executable.

[0039] The application building and deployment module 400 connects to the user interaction subsystem 100 at its input end and to the end-user device (mobile phone / tablet) and the actual robot controller at its output end. It is used to convert flowcharts into executable code for the robot and end-user applications.

[0040] The application building and deployment module 400 includes a parameter parser, an H5 generator, and a script compiler. The parameter parser automatically identifies "configurable variables" in the flowchart and extracts them as form items. The H5 generator automatically encapsulates the form items into an HTML5 mobile application interface. The script compiler compiles general flowcharts into low-level code executable by a specific brand of robot, such as Ybot R2 or Ybot M200 in this embodiment, such as G-code or vendor-specific scripts.

[0041] Compared to traditional teaching programs, which are often deeply tied to specific robot brands and whose code is difficult to reuse.

[0042] This invention uses a script compiler to map standardized intermediate instructions to the underlying code of specific models (such as Ybot R2 or other brands), enabling the same set of natural language-generated business logic to be easily deployed on hardware of different configurations (robotic arms, humanoids, and composite robots) or different brands, thereby enhancing the reusability and flexibility of the system.

[0043] This embodiment uses "developer's super-automatic inventory management application" as an example for illustration.

[0044] like Figure 2 As shown, the embodied intelligence application software development method based on multi-agent collaboration, applying the aforementioned embodied intelligence application software development system based on multi-agent collaboration, includes the following steps: Step S1: Intent Input and Intent Parsing.

[0045] Natural language commands are input through the user interaction subsystem 100, and the scenario-defined intelligent agent 201 parses the executing subject, action intent, and target object in the natural language commands to generate an abstract task sequence. For example... Figure 3 As shown.

[0046] In this embodiment, the scene definition agent 201 identifies two execution entities: Robot A: Ybot R2, Robot B: Ybot M200 and four key action nodes: Pick (box) -> Place (table) -> Cut (box) -> Pick (product) -> Place (shelf). At this time, the generated node parameters are empty (coordinates unknown).

[0047] Step S2: Environmental parameter fusion and pose calculation.

[0048] The environment-aware intelligent agent 202 queries the simulation engine 300 based on the target object in the abstract task sequence to obtain the physical attribute data of the target object in the virtual scene. The environment-aware intelligent agent 202 further calculates and generates the working pose and motion path of the robot's end effector relative to the target object based on the preset robot kinematics model and physical attribute data, and maps the physical attribute data to the abstract task sequence to generate a parameterized task sequence.

[0049] In this embodiment, specifically: the environment-aware agent 202 receives the output of the scene-defining agent 201 and executes the following logic: Upon parsing the keyword "box", a query command QueryObject("box") is sent to the simulation engine 300. The simulation engine 300 returns a list of all "box" entities in the scene, and the environment-aware agent 202 maintains a semantic-physical mapping table. When the parsed object is "box", it automatically queries all instances labeled "box" in the simulation engine 300 and selects the best match based on the task context (such as "nearest box"), filling its pose data into the parameter slots of the action node. For example: {id: "box_001", pos: [1.5, 0.5, 0], size: [0.3, 0.3, 0.3]}; the environment-aware agent 202 automatically calculates the joint angle (Pose) required for the robot to grasp this position and fills target_pose=[1.5, 0.5, 0] into the Pick node. Similarly, the coordinate parameters of "table" and "shelf" are automatically obtained.

[0050] Step S3: Visual flowchart generation and arrangement.

[0051] The visual intelligent agent 203 converts the parameterized task sequence into a visual task flowchart containing logical connections, and renders it in the user interaction subsystem 100. For example... Figure 4 As shown.

[0052] In this embodiment, the parameterized node sequence is converted into graphical primitives. The system automatically inserts necessary logical judgment nodes (e.g., inserting a VisualCheck node before "open the box" to confirm whether the box is in place). The user can then visually see the logical structure and make drag-and-drop adjustments.

[0053] Step S4: Interactive incremental update.

[0054] In response to user-initiated modification commands to the visual task flowchart via natural language or graphical interaction, the environment-aware agent 202 identifies and updates only the environmental parameters of the task nodes associated with the modified content. Pre-computed paths (such as collision-free paths) associated with the updated nodes are marked as 'invalid' and trigger replanning during the final code compilation phase to ensure physical correctness. The visual agent 203 refreshes the visual task flowchart in real time; this real-time refresh includes not only updating interface parameters but also marking the validity of the backend path planning cache. Figure 5 As shown.

[0055] In this embodiment, specifically: the user finds that the default placement in the flowchart is on the 1st shelf, so he / she inputs: "Change the placement position to the 3rd shelf"; the environmental perception agent 202 queries the height data of "3rd shelf" in the simulation environment again (e.g., z=1.2m), and the visualization agent 203 only updates the parameters of the Place (shelf) node, while the rest of the flowchart remains unchanged.

[0056] Step S5: Application packaging and real device operation.

[0057] The application build and deployment module 400 is used for script compilation, application generation, and delivery. For example... Figure 6 As shown.

[0058] In this embodiment, specifically: when the user clicks "Publish Application", the application building and deployment module 400 performs the following actions: Script compilation: Convert the flowchart into robot control code and distribute it to the actual robot controller IP via the network.

[0059] UI generation: Identify "product type" and "target shelf" as variables to generate a simple app interface.

[0060] Delivery: A QR code is displayed on the screen. After scanning the code, the user clicks "Start" on the mobile app, and the real robot then performs the sorting task according to the planned path.

[0061] Through the above steps, the specific process of this embodiment is as follows: The user (such as a supermarket store manager) inputs the natural language command "develop a stocking application" in the user interaction subsystem 100. The Ybot R2 robot moves the box on the ground to the table, and then the Ybot M200 robot cuts open the box with a knife and takes out the goods inside and puts them on the shelf.

[0062] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely prisms of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A software development system for embodied intelligence applications based on multi-agent collaboration, characterized in that: It includes a user interaction subsystem (100), a multi-agent collaborative orchestration center (200), a simulation runtime engine (300), and an application building and deployment module (400), among which: The user interaction subsystem (100) provides a natural language dialogue window and a visual flowchart editor; A multi-agent collaborative orchestration center (200), connected to a user interaction subsystem (100), includes: A scenario-defined intelligent agent (201) is used to parse natural language instructions input via the user interaction subsystem (100) and generate an abstract task sequence containing the executing subject, action intention and target object; An environment-aware intelligent agent (202) is connected to a scene-defining intelligent agent (201) and a simulation running engine (300) respectively. It is used to query the simulation running engine (300) based on the target object in the abstract task sequence to obtain the physical attribute data of the target object in the virtual environment, and to map the physical attribute data to the abstract task sequence to generate a parameterized task sequence. The visual intelligent agent (203) is connected to the user interaction subsystem (100) and the environment-aware intelligent agent (202) to convert the parameterized task sequence into a visual task flowchart and render it on the user interaction subsystem (100); The simulation runtime engine (300) is used to respond to query requests from the environment-aware agent (202) and provide physical parameters in the virtual environment; The application building and deployment module (400) is connected to the user interaction subsystem (100) and is used to convert the visual task flowchart into robot executable code and end-user applications.

2. The embodied intelligence application software development system based on multi-agent collaboration according to claim 1, characterized in that: The user interaction subsystem (100) includes a multimodal input interface (101) and a real-time feedback module (102) connected to the multimodal input interface (101). The multimodal input interface (101) integrates a chat window and a visual flowchart editor.

3. The embodied intelligence application software development system based on multi-agent collaboration according to claim 1, characterized in that: The simulation runtime engine (300) includes a scene graph database and a physical simulation kernel.

4. The embodied intelligence application software development system based on multi-agent collaboration according to claim 1, characterized in that: The application building and deployment module (400) includes a parameter parser, an H5 generator, and a script compiler.

5. A method for developing embodied intelligence application software based on multi-agent collaboration, characterized in that: The embodied intelligence application software development system based on multi-agent collaboration as described in any one of claims 1 to 4 includes the following steps: Step S1, Intent Input and Intent Resolution: Users input natural language commands through the user interaction subsystem (100), and the scene-defined intelligent agent (201) parses the execution subject, action intention and target object in the natural language commands to generate an abstract task sequence; Step S2, Environmental Parameter Fusion and Pose Calculation: The environment-aware intelligent agent (202) queries the simulation engine (300) based on the target object in the abstract task sequence to obtain the physical attribute data of the target object in the virtual scene. The environment-aware intelligent agent (202) further calculates and generates the working pose and motion path of the robot's end effector relative to the target object based on the preset robot kinematics model and physical attribute data, and maps the physical attribute data to the abstract task sequence to generate a parameterized task sequence. Step S3: Visual flowchart generation and arrangement: The visual agent (203) converts the parameterized task sequence into a visual task flowchart containing logical connections and renders it in the user interaction subsystem (100); Step S4, Interactive Incremental Update: In response to the user’s modification command to the visual task flowchart through natural language or graphical interaction, the environment-aware agent (202) identifies and updates only the environmental parameters of the task nodes associated with the modified content. The pre-computed path associated with the updated node will be marked as 'invalid' and will be replanned during the final code compilation stage. The visual agent (203) refreshes the visual task flowchart, updates the interface parameters, and marks the validity of the backend path planning cache in real time. Step S5: Application packaging and real device operation; The application build and deployment module (400) is used to compile scripts, generate applications, and deliver them for use.

6. The method for developing embodied intelligent application software based on multi-agent collaboration according to claim 5, characterized in that: The physical attribute data in step S2 includes at least: the unique identifier of the target object, its three-dimensional spatial coordinates, three-dimensional dimensions, and Euler angles.

7. The method for developing embodied intelligent application software based on multi-agent collaboration according to claim 5, characterized in that: In step S5, the script compilation specifically involves compiling the task flowchart into robot control code and sending it to the target controller for execution.

8. The method for developing embodied intelligence application software based on multi-agent collaboration according to claim 5, characterized in that: In step S5, the application generation specifically involves: parsing the configurable parameter nodes in the task flowchart, and automatically generating a graphical user interface or H5 mobile application containing human-computer interaction controls based on the configurable parameter nodes.

Images