A control method and system for a robot

Abstract

This application provides a robot control method and system, relating to the field of artificial intelligence technology. In this method, m first interface information points from the ROS (Real-Time Optical System) are obtained, where m is a positive integer. For each of the m first interface information points, an interface definition file is obtained based on the type of the first interface information. The interface definition file is recursively parsed based on the AST (Abstract Syntax Tree) to obtain corresponding annotation information and parameters. Semantic information is obtained from the annotation information based on NLP (Natural Language Processing), which includes a detailed description of the function and parameters of the first interface information. Based on the interface name, semantic information, and parameters of the first interface information, p second interface information points conforming to MCP (Multi-Channel Programming) are generated, where p is a positive integer. In response to a user's first natural language instruction, a first operation command is generated based on LLM (Limited Language Modeling) and the p second interface information points. Based on ROS and MCP, the robot is instructed to execute the first operation command, obtaining first return information for displaying the result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot control method and system. Background Technology

[0002] In recent years, with the rapid development of robotics technology, Robot Operating System (ROS) has become a widely adopted middleware framework in academia and industry. ROS provides a rich set of interfaces designed to simplify the control of robots.

[0003] The combination of Model Context Protocol (MCP) and Large Language Model (LLM) has been widely used in robot development. LLM uses the understanding of user's natural language commands to control the robot based on MCP and ROS. However, the interface provided by ROS is based on data structure definitions, making it difficult to associate with complex user commands. Therefore, LLM can only understand simple and precise user commands, and struggles to understand complex or ambiguous ones. Summary of the Invention

[0004] In view of this, this application provides a robot control method and system to improve the robot's ability to understand user natural language commands.

[0005] Firstly, this application provides a robot control method, which, by way of example, can be applied to a robot's control system. The method includes the following processes:

[0006] Obtain information about m first interfaces in ROS, where m is a positive integer. Each first interface includes the interface name and interface type.

[0007] Based on the interface type of the i-th first interface information among the above m first interface information, obtain the interface definition file corresponding to the i-th first interface information, where i is a positive integer and can be any value between 1 and m.

[0008] Based on the Abstract Syntax Tree (AST), the interface definition file is recursively parsed to obtain the annotation information corresponding to the i-th first interface information and... One parameter, Each parameter is a natural number, and each parameter includes a parameter name and a data type.

[0009] Based on Natural Language Processing (NLP) technology, semantic information corresponding to the i-th first interface information is obtained from the above annotation information. This semantic information includes the interface function description corresponding to the i-th first interface information. When there is at least one parameter, the semantic information also includes at least one of the following: parameter function description, physical meaning, unit, constraint condition and default value corresponding to at least one parameter.

[0010] Based on the interface name of the i-th first interface information mentioned above, and the semantic information corresponding to the i-th first interface information, and The parameters are used to generate at least one second interface information that conforms to MCP, where the at least one second interface information corresponds to the interface name, semantic information, and... of the i-th first interface information. The parameters are corresponding to There are *m* parameters, where *m* first interface information pieces correspond to *p* second interface information pieces, where *p* is a positive integer greater than or equal to *m*. It is a natural number.

[0011] In response to the user's first natural language instruction, a first operation command is generated based on the Large Language Model (LLM) and p second interface information. This first operation command corresponds to q of the p second interface information, where q is a positive integer and q is less than or equal to p.

[0012] Based on ROS and MCP, at least one robot is instructed to execute a first operation command and obtain first return information, which is used to display the operation result corresponding to the first natural language instruction.

[0013] Secondly, this application provides a control system for a robot, the system including a processing module that can implement the steps of the method as described in the first aspect.

[0014] Thirdly, this application provides an electronic device including a memory and a processor, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and one or more processors calling the computer instructions to cause the electronic device to perform the steps of the method as described in the first aspect.

[0015] Fourthly, this application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the steps of the method as described in the first aspect.

[0016] Fifthly, this application provides a program product including instructions that, when executed on an electronic device, cause the electronic device to perform the steps of the method as described in the first aspect.

[0017] In a sixth aspect, this application provides a chip including a processor coupled to a memory, the processor being configured to execute a computer program or instructions stored in the memory, such that the chip implements the steps of the method described in the first aspect.

[0018] In this embodiment, the first interface information of ROS is first obtained, and the interface definition file is obtained according to the interface type. Then, the interface definition file is parsed based on AST to obtain the annotation information and parameters corresponding to the first interface information, and semantic information is obtained from the annotation information based on NLP. The semantic information includes the interface function description and detailed parameter information, which can provide a detailed functional explanation of the first interface information. Subsequently, second interface information corresponding to the interface name, semantic information and parameters of the first interface information is generated. In response to the user's natural language command, an operation command is generated based on LLM and the second interface information. Finally, the robot is instructed to execute the operation command based on ROS and MCP to obtain return information to display the operation result. Since the semantic information corresponding to the second interface information is obtained from the annotation information of the ROS interface definition file based on NLP, this semantic information can provide a detailed functional explanation of the first interface information. Therefore, LLM can accurately understand the interface function of ROS based on the second interface information, improve the understanding of the user's natural language command, and thus generate accurate operation commands, thereby reducing the difficulty of robot control.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the operation of a robot based on LLM, MCP, and ROS.

[0021] Figure 2 A flowchart illustrating the robot control method provided in an embodiment of this application;

[0022] Figure 3 This is another schematic flowchart of the robot control method provided in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the robot state in the robot control method provided in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram of the capability mapping process based on static configuration in the robot control method provided in the embodiments of this application;

[0025] Figure 6This is a schematic diagram of the safety inspection process in the robot control method provided in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram illustrating the performance optimization of the robot control method provided in the embodiments of this application;

[0027] Figure 8 A schematic diagram illustrating the module architecture and data flow of the robot control system provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the module interaction of the robot control system provided in an embodiment of this application. Detailed Implementation

[0029] To ensure clarity and conciseness in the description of the following embodiments, the terminology used in the embodiments of this application will first be explained. It should be understood that this explanation is for the purpose of better understanding the embodiments of this application and does not necessarily constitute a limitation on the embodiments of this application.

[0030] The Robot Operating System (ROS) is an open-source meta-operating system. Its core idea is to organize a robot's functionality into multiple nodes, which communicate with each other by sending messages. ROS includes hardware abstraction, low-level driver management, execution of common functions, inter-program message passing, and program distribution package management. It also provides tools and libraries for acquiring, building, writing, and executing multi-robot fusion programs. In 2017, ROS developers released the second version of the Robot Operating System, ROS2. For clarity, the first major series of ROS versions will be referred to as ROS1 below.

[0031] An Abstract Syntax Tree (AST) is an abstract representation of the syntactic structure of source code. It represents the syntactic structure of a programming language in a tree-like structure, where each node represents a structure within the source code. The AST is an important intermediate step for compilers and interpreters when processing source code, aiding in code analysis, optimization, and transformation.

[0032] Natural Language Processing (NLP) is an important research area in artificial intelligence. It is an interdisciplinary field integrating computer science, artificial intelligence, and linguistics, encompassing two main aspects: natural language understanding and natural language generation. Its research covers multiple levels, including characters, words, phrases, sentences, paragraphs, and texts, serving as a bridge between machine language and human language. It aims to enable machines to understand, interpret, and generate human language, achieving effective communication between humans and machines, and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0033] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data. These models can generate natural language text or understand the meaning of language text. Their core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent. LLMs can perform not only simple language tasks such as spell checking and grammar correction, but also complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation.

[0034] The Model Context Protocol (MCP) is an open-source protocol designed to integrate LLMs with external data sources and tools, establishing a secure, bidirectional connection between the LLM and the data source. This protocol handles both local and remote resources using the same protocol. MCP includes common communication protocols, data formats, and rules, offering features such as simplified development, flexibility, real-time response, security and compliance, and scalability. The protocol employs a client-server architecture, supporting three functional types: tools, resources, and prompts. Host applications can connect to multiple servers through standardized interfaces to obtain context data understandable by the LLM.

[0035] In recent years, with the rapid development of robotics technology, ROS has become a widely adopted middleware framework in academia and industry. ROS provides a rich set of interfaces designed to simplify the control of robots.

[0036] In ROS, the basic unit of an executable program is a node, which is used to operate the robot. Nodes communicate with each other through messages. The message mechanism includes services, actions, and topics. Developers use these three message mechanisms to manipulate nodes, and thus control the robot. The following describes these three message mechanisms:

[0037] Mechanism 1: Topics are an asynchronous, streaming messaging mechanism. Robot nodes publish status information to specific topics at a fixed frequency, and other nodes can obtain the latest data in real time by subscribing to these topics. Topics are suitable for continuously updated status data, such as frequently updated statuses or continuous data streams that need to be monitored in real time.

[0038] Mechanism 2, services, is a synchronous request-response messaging mechanism. When a certain status needs to be queried, the service provided by the robot node can be called, and the robot node immediately returns the current status. Services are suitable for low-frequency queries, such as getting the current battery percentage, getting the current pose that does not change frequently, and triggering a one-time status report.

[0039] Mechanism 3, actions, are an asynchronous, feedback-based messaging mechanism often used to execute complex or long-term tasks. During task execution, the action service can periodically publish feedback. For example, when performing a navigation task, it can obtain the robot's current position in real time.

[0040] In traditional robot control processes, developers or operators typically rely on command-line interfaces (CLIs) such as ROS2 topics and services, or specialized graphical user interfaces (GUIs) like Rviz, RQt, and Gazebo, to control robots. However, CLI-based robot control requires operators to be proficient in ROS communication mechanisms and message definitions, presenting a high technical barrier. GUI-based control, on the other hand, relies on statically configured graphical tools, resulting in rigid interaction methods that struggle to adapt flexibly to dynamic changes in robot functionality. Furthermore, both of these methods rely on control interfaces based on strictly defined data structures, making integration with large language models difficult. They lack understanding and adaptability to ROS interface semantics, further complicating robot control.

[0041] In addition to the control methods mentioned above, external systems can also control robots through interfaces, typically using the following three methods:

[0042] Method 1 is a WebSocket communication method based on the ROS bridging protocol (rosbridge). This method establishes a WebSocket connection between ROS and the external system through the rosbridge server, transmitting ROS messages in JavaScript Object Notation (JSON) format. The external system subscribes to topics or calls services by sending JSON strings via WebSocket.

[0043] Method 2 is a communication method based on the Representational State Transfer (REST) ​​Application Programming Interface (API). This method develops a web server layer on top of the ROS node, encapsulating specific ROS functions as Hypertext Transfer Protocol (HTTP) interfaces. External systems can then write code to map ROS functions to the API and control the robot via HTTP.

[0044] Method 3, robot control based on MCP and LLM. This method introduces MCP as a standardized intermediate layer between ROS and LLM. Through the understanding of the user's natural language commands by LLM, operating commands for ROS are generated, and robot control is achieved based on MCP and ROS.

[0045] The first two methods mentioned above require complex development of the ROS interface when implementing external system control of the robot. They lack understanding of the semantics of the ROS interface and the ability to adapt, making robot control very difficult.

[0046] With the rapid development of artificial intelligence and LLM, method 3 mentioned above has been widely applied. The following section combines... Figure 1 Method 3 described above will be described in detail.

[0047] like Figure 1 As shown, ROS contains multiple robot nodes. Nodes operate on specific robots, and each node can perform at least one specific robot operation. Nodes provide message interfaces to higher-level robots; each node can provide at least one message interface, and these message interfaces use communication mechanisms such as services, actions, or topics.

[0048] MCP is an open-source license for integrating large language models (LLMs) with external data sources or tools. This license employs a client-server architecture. The MCP Client acts as a bridge between the user, the LLM, and the MCP Server. The MCP Server supports three functional types: tools, resources, and prompts. These three functional types are described below:

[0049] Type 1, resource, is file-like data that can be read by the MCP Client.

[0050] Type 2, tools, are functions or methods that can be called.

[0051] Type 3, prompts, are pre-written templates that help users complete specific tasks.

[0052] The basic process of controlling a robot based on MCP and LLM is as follows:

[0053] Step 1: The MCP Server registers the ROS interface as a tool and sends a list of available tools to the MCPClient;

[0054] Step 2: The MCP Client responds to the user's operation instructions and sends the operation instructions and a list of available tools to the LLM;

[0055] Step 3: The LLM interprets user commands and tells the MCP Client which tools to use;

[0056] Step 4: The MCP Client executes the corresponding tool calls through the MCP Server;

[0057] Step 5: ROS uses this tool to perform operations on the robot and returns the results to the MCP Server;

[0058] Step 6: The MCP Server returns the results to the MCP Client.

[0059] Step 7: The MCP Client displays the results to the user.

[0060] In the above method, the ROS interface is defined based on a data structure. When the MCP Server registers the ROS interface as a tool, it registers the ROS interface name and the required parameters together. This allows the LLM to understand the ROS interface functionality through the interface name. Then, the LLM understands the user's operation commands, selects the appropriate tool from the MCP tool list, and informs the MCP Client. Finally, the MCP Server executes the tool call, which in turn triggers the corresponding ROS interface call, thus achieving robot control.

[0061] However, ROS interface names are often complex words or combinations of word roots, making them difficult to understand accurately. It's challenging to precisely associate the interface's function with complex natural language commands simply by its name. Therefore, LLMs can only understand simple and precise user commands; they struggle with complex or ambiguous commands, thus failing to accurately select the appropriate tools to control the robot.

[0062] In view of this, this application provides a robot control method and system to improve the understanding of user natural language commands. The method provided in the embodiments of this application can be applied to the control system of a robot.

[0063] Next, combined Figures 2 to 7 This application provides a detailed description of the robot control method provided in the embodiments of this application, which is applicable to ROS1 and ROS2.

[0064] Example 1: As Figure 2 As shown in the figure, this application proposes a robot control method that can be applied to the robot's control system.

[0065] This method first obtains the first interface information of ROS, including the interface name and interface type, and then retrieves the interface definition file based on the interface type. Next, the interface definition file is parsed based on AST to obtain the annotation information and parameters corresponding to the first interface information. Semantic information is then obtained from the annotation information using NLP, which includes a description of the interface function and detailed parameter information, providing a comprehensive functional explanation of the first interface information. Subsequently, second interface information corresponding to the aforementioned interface name, semantic information, and parameters is generated. Responding to the user's natural language commands, operation commands are generated based on LLM and the second interface information. Finally, the robot is instructed to execute the operation commands based on ROS and MCP, and return information is obtained to display the operation results.

[0066] Using the above method, the semantic information of the interface can be obtained from the comments in the ROS interface definition file. This semantic information can provide a detailed functional explanation of the first interface information, enabling the LLM to accurately understand the ROS interface function, rather than just guessing based on the name and type. This can improve the LLM's ability to understand user natural language instructions, thereby generating accurate operation commands and reducing the difficulty of robot control.

[0067] The following is combined with Figure 2 The specific steps of this method are described in detail.

[0068] S201, obtain m first interface information of ROS, where m is a positive integer.

[0069] The first interface information here is the interface information of ROS that provides message services to the outside world. The first interface information includes the interface name (such as / cmd_vel) and the interface type (such as geometry_msgs / Twist).

[0070] For ROS1, interface information can be obtained through the ROS Master's API or through Remote Procedure Call (RPC) methods based on Extensible Markup Language (XML). Interface information can also be obtained through operation commands such as "rostopic", "rosservice", "rosmsg", and "rossrv".

[0071] For ROS2, computation graph information can be scanned using the Data Distribution Service (DDS) Discovery protocol or ROS2's core APIs (such as get_service_list and get_topic_list). The computation graph is a network structure in ROS2 where nodes communicate via messages. Scanning the computation graph allows you to obtain ROS2's interface information.

[0072] S202, Obtain the interface definition file based on the interface type. Based on the interface type of the i-th first interface information among the m first interface information, obtain the interface definition file corresponding to the i-th first interface information, where i is a positive integer and can be any value between 1 and m.

[0073] Each interface in ROS corresponds to an interface definition file, typically a .msg or .srv file. These files can be found in ROS by interface type.

[0074] S203, based on AST, recursively parses the interface definition file to obtain the annotation information corresponding to the i-th first interface information and... One parameter, The parameter is a natural number, and the parameter includes the parameter name and data type.

[0075] AST uses regular expression matching to parse the data source into a sequence of smallest units, and then converts the sequence of smallest units into a tree structure according to syntax rules.

[0076] In this step, the interface definition file is matched line by line using regular expressions. If the interface contains at least one parameter, the parameter name (e.g., linear) and data type (e.g., float32) are parsed. The interface name and parameters are then converted into a tree-structured interface information based on the syntax nesting.

[0077] For special data types, recursion is required for parsing. For example, for array type parameters, the parsed element type is set to array by recognizing the suffixes "[" and "]", and the array member types are then recursively parsed and populated into items. For custom message types in ROS, such as geometry_msgs / Vector3, the parsing results are embedded as sub-objects into the properties of the current element through recursive calls.

[0078] Simultaneously, the wildcard "#" is used to parse the comment information in the interface definition file. Optionally, the comment information may include block comments and / or inline comments.

[0079] Optionally, before recursively parsing the interface definition file based on the AST, a collection of visited types (visited_types) can be maintained. Before each recursive step, it is first checked whether the type of the current parameter is already in this collection. If it exists, a $ref reference (JSON pointer) pointing to the type definition is directly generated instead of continuing the recursive expansion. This mechanism can effectively prevent infinite recursion of the tree structure from causing a stack overflow.

[0080] In some implementations, blank lines and irrelevant whitespace characters can be removed before recursively parsing the interface definition file, and then the remaining content can be recursively parsed. This can reduce the impact of invalid characters on recursive parsing and improve the efficiency of recursive parsing.

[0081] S204, based on NLP, obtain the semantic information corresponding to the first interface information from the annotation information. This semantic information includes the interface function description corresponding to the i-th first interface information. When there is at least one parameter, the semantic information also includes at least one of the following: parameter function description, physical meaning, unit, constraint condition and default value corresponding to at least one parameter.

[0082] NLP uses techniques such as text preprocessing, syntactic analysis, and semantic analysis to help computers better understand and process natural language.

[0083] Understandably, annotation information typically contains detailed descriptions of interface information. Therefore, analyzing annotation information using NLP techniques can yield semantic information about the interface, such as the interface function description, the physical meaning of parameters, units (e.g., "m / s"), and constraints (e.g., "0~100"). This semantic information helps LLM accurately understand the ROS interface functionality, including the understanding of parameters, rather than simply guessing based on names and types.

[0084] For example, without semantic information, the LLM only knows the name and data type of the parameter "theta". If the semantic information tells the LLM that this parameter represents "target orientation", the unit is "radians", and the constraint is "the value range is within -3.14 to 3.14", then the LLM can truly understand the specific meaning of this parameter and, in subsequent steps, understand the user's natural language instructions and generate accurate operation commands.

[0085] S205, based on the interface name, semantic information, and... corresponding to the i-th first interface information The parameters are used to generate at least one second interface information that conforms to MCP, where the at least one second interface information corresponds to the interface name, semantic information, and... of the i-th first interface information. The parameters are corresponding to One parameter, It is a natural number.

[0086] In some implementations, the second interface information can be in JSON schema format. All parsed fields are assembled into a standard JSON schema object. Required attributes are automatically tagged according to the field definitions, and the extracted semantic descriptions are injected into schema keywords such as description, type, properties, minimum, and maximum, ultimately generating complete second interface information. JSON schema is a general intermediate representation supported by MCP. This intermediate representation recursively describes each field in a nested object format, facilitating understanding by LLM.

[0087] Through the above steps S202 to S205, at least one second interface information corresponding to each of the m first interface information can be obtained, and the m first interface information corresponds to p second interface information, where p is a positive integer and p is greater than or equal to m.

[0088] It is understandable that one first interface information usually corresponds to one second interface information. However, in some implementations, one first interface information can correspond to multiple second interface information. For example, for a first interface information with a message mechanism of action, it can correspond to two second interface information, used to trigger a task and query results, respectively.

[0089] S206, in response to the user's first natural language instruction, generates a first operation command based on the LLM and p second interface information. The first operation command corresponds to q of the p second interface information, where q is a positive integer and q is less than or equal to p.

[0090] LLM understands the function of the ROS interface through the semantic information in the second interface information, then understands the user's first natural language instruction, selects q second interface information from p second interface information generated in the previous steps, and generates the corresponding first operation command.

[0091] S207, based on ROS and MCP, instruct at least one robot to execute a first operation command and obtain first return information, which is used to display the operation result corresponding to the first natural language instruction.

[0092] One implementation involves the MCP Client sending the first operation command obtained in step S206 to the MCP Server. The MCP Server then calls the corresponding ROS message interface, instructing the ROS nodes to execute the operation command on at least one robot. The MCP Server then sends the return information obtained through the ROS message interface to the MCP Client, which in turn displays the final first return information to the user.

[0093] In some implementations, for steps S201 to S205 above, the ROS can be actively scanned during system startup to obtain the first interface information and generate the second interface information through parsing. Alternatively, during system operation, the reflection mechanism of ROS can be used to dynamically obtain newly added interfaces in ROS, obtain the latest first interface information in real time, and update the second interface information promptly after parsing.

[0094] Example 2, as Figure 3 As shown in the figure, this application proposes a robot control method, which can be applied to the robot's control system. Figure 2 Compared to Example 1, Example 2 adds features such as robot state machine and MCP function registration, three-layer safety filtering mechanism, multimodal data optimization processing, visual feedback and complete interactive closed loop.

[0095] This method first obtains the first interface information of ROS, including services, actions, and topics. A first-layer security filter is then applied to this first interface information based on a blacklist and whitelist to determine whether its use is permitted. For permitted first interface information, the interface definition file is obtained according to its interface type. The interface definition file is then parsed using an Abstract Syntax Tree (AST) to obtain the corresponding annotation information and parameters. Semantic information is then extracted from the annotation information using Natural Language Processing (NLP). This semantic information includes a description of the interface function and detailed parameter information, providing a comprehensive functional explanation of the first interface information.

[0096] Then, second interface information corresponding to the interface name, semantic information, and parameters of the first interface information is generated. The second interface information corresponding to services and actions is registered as an MCP tool, and the second interface information corresponding to topics is registered as an MCP resource. Robot state information, such as current state, task state, and environmental information, is obtained through ROS nodes, and the robot's state is managed using a state machine model. The robot's state information is then registered as MCP prompts.

[0097] Responding to the user's natural language commands, the robot obtains status prompts through the MCP (Multi-Channel Programming) system. The LLM (Limited Linear Modulation) system understands the user commands and the robot's status prompts, and generates operation commands based on the tools and resources of the MCP. A second layer of safety filtering is applied to the operation commands based on parameter constraints, filtering out commands with incorrect parameters. A third layer of safety filtering is then applied to the operation commands through human-machine loopback confirmation for high-risk operations.

[0098] Finally, based on ROS and MCP, the robot executes operation commands. Using MCP tools and resources, the returned information is optimized through methods such as rate limiting, compression, and downsampling, and then a visualized operation result is presented to the user. This visualized result serves as the basis for the next round of user-robot interaction, forming a complete and sustainable interaction chain.

[0099] Using the above method, semantic information of the interface can be obtained from the comments in the ROS interface definition file. This semantic information can provide a detailed functional explanation of the first interface information, enabling LLM to accurately understand the ROS interface functions instead of guessing based solely on names and types. This allows for the understanding of complex or ambiguous user commands, and combined with robot status prompts, more accurate operation commands can be generated, further reducing the difficulty of robot operation. By registering MCP tools, resources, and prompts, the ROS interface and robot status are mapped to various MCP functions, making robot control more convenient. A three-layer safety filtering mechanism enables safety management at each stage of robot control, improving the safety of robot control. Optimization processes such as rate limiting, compression, and downsampling make the processing of returned information more efficient. Using the visualization results as the basis for the next round of interaction forms a complete and sustainable interaction chain, enabling precise multi-round control of the robot.

[0100] In summary, by adopting the above methods, LLM can accurately understand the interface functions of ROS, improve its ability to understand user natural language commands, and make robot control simpler, safer, more convenient and efficient.

[0101] The following is combined with Figure 3 The specific steps of this method are described in detail.

[0102] S301, Device starts up, initializing ROS connection.

[0103] Start the MCP Server and load the ROS environment variables. Establish an asynchronous event loop to prepare for handling ROS interface callbacks.

[0104] S302, Scan ROS to obtain information on n first interfaces of ROS, where n is a positive integer and is greater than or equal to m.

[0105] The method for obtaining ROS interface information in this step is the same as... Figure 2 The method used in step S201 of the illustrated embodiment is the same.

[0106] In addition, ROS provides interfaces for three messaging mechanisms: service, action, and topic. Therefore, the first interface message obtained here includes the interfaces corresponding to these three messaging mechanisms.

[0107] S303, based on at least one of the following, perform security filtering on n pieces of first interface information to obtain m pieces of first interface information:

[0108] Based on the established whitelist of first operation interfaces, security filtering is performed on n pieces of first interface information, and the resulting m pieces of first interface information are all in the whitelist of first operation interfaces.

[0109] Based on the established blacklist of first operation interfaces, security filtering is performed on n pieces of first interface information, and the resulting m pieces of first interface information are all not in the first operation interface blacklist.

[0110] In this step, the interface information is filtered for the first layer of security through whitelists and / or blacklists, ensuring that the filtered interface information is within the allowed scope of use during the ROS interface capability discovery phase.

[0111] Optionally, the first blacklist of operation interfaces here can include interfaces internal to the ROS system, such as the system log interface / rosout, which is a global topic enabled by default in ROS. The blacklist can also include high-risk interfaces, such as the format interface / format_disk, whose operation may lead to serious consequences.

[0112] S304, Obtain the interface definition file based on the interface type.

[0113] This step and Figure 2 Step S202 of Embodiment 1 is the same and will not be repeated here.

[0114] S305 parses the interface definition file based on AST.

[0115] This step and Figure 2 Step S203 of Embodiment 1 shown is the same and will not be repeated here.

[0116] S306, this step includes two operations: type mapping conversion and extraction of annotation semantics.

[0117] The content of extracting annotation semantics and Figure 2 The steps S204 of the illustrated embodiment are the same and will not be repeated here. Only the content of type mapping conversion will be described.

[0118] Because ROS and MCP define parameter data types differently, it's necessary to convert the ROS interface parameter data types to MCP-compliant data types. Therefore, after parsing the interface definition file based on the AST, the obtained ROS data types need to be mapped to MCP-compliant data types, which typically use JSONSchema data types.

[0119] The specific mapping rules are shown in Table 1 below:

[0120] ROS data types JSON Schema data types Constraints bool (Boolean) boolean int8-int64 (integers) integer minimum, maximum float32, float64 (floating-point numbers) number (numerical value) minimum, maximum string (string) string pattern (regular expression) time (duration) object {secs (second-level timestamp), nsecs (nanosecond offset)} T[] (array) array items:T_Schema

[0121] Table 1

[0122] For example, parameters in ROS with integer data types, including int8, int16, int32, and int64, are mapped to the integer data type in the JSON Schema, and minimum and maximum value constraints are defined, i.e., the range of values ​​for the parameter. When extracting annotation semantic information, if the parameter constraints are obtained, the mapping is performed according to the constraint format in the table above.

[0123] In some implementations, the type mapping conversion and the extraction of annotation semantics can be performed simultaneously; alternatively, the type mapping conversion can be performed first, followed by the extraction of annotation semantics; and still others, the extraction of annotation semantics can be performed first, followed by the type mapping conversion. This application does not limit the execution order of the above two operations.

[0124] S307, Generate the second interface information.

[0125] After completing the type mapping conversion and extracting annotation semantics in step S306 above, the result information is used to generate p second interface information in JSON Schema format.

[0126] The following example demonstrates a ROS navigation service interface and a comparison of its converted JSON Schema:

[0127] The contents of the interface definition file nav_msgs / srv / SetGoal.srv obtained in step S304 above are as follows:

[0128] # Set robot navigation target point

[0129] # Target location (meters)

[0130] geometry_msgs / Point position

[0131] # Target orientation (radians, -3.14 ~ 3.14)

[0132] float32 theta

[0133] ---

[0134] # Success or failure?

[0135] bool success

[0136] # error message

[0137] string message

[0138] After parsing, the second interface information in JSON Schema format is as follows:

[0139] {

[0140] "name": "nav_set_goal",

[0141] "description": "Setting the robot's navigation target point",

[0142] "inputSchema": {

[0143] "type": "object",

[0144] "properties": {

[0145] "position": {

[0146] "type": "object",

[0147] "description": "Target location (meters)",

[0148] "properties": {

[0149] "x": { "type": "number"},

[0150] "y": { "type": "number"},

[0151] "z": { "type": "number"}

[0152] },

[0153] "required": ["x", "y"]

[0154] },

[0155] "theta": {

[0156] "type": "number",

[0157] "description": "Target orientation (radians)",

[0158] "minimum": -3.14,

[0159] "maximum": 3.14

[0160] }

[0161] },

[0162] "required": ["position", "theta"]

[0163] }

[0164] }

[0165] As can be seen from the above comparison, the above steps not only convert the data type float32 into the data type number, but also extract the comment information in the interface definition file into the description field of the second interface information, and convert the value range of this parameter (-3.14~3.14) into minimum and maximum format constraints, thereby achieving complete semantic preservation, enabling LLM to more accurately understand the interface functions of ROS.

[0166] In some implementations, steps S303 to S307 above can also employ the following method to generate second interface information based on the interface definition file. The content of the interface definition file is directly used as a prompt for the MCP and sent to the LLM, allowing the LLM to generate second interface information in JSON Schema format. This method can handle very complex or non-standard comment formats, and the generated descriptions are more in line with natural language, making it suitable for generating auxiliary tools during the development phase.

[0167] S308, Register MCP functionality, including tools, resources, and tips.

[0168] If the message mechanism corresponding to the i-th first interface information among the m first interface information is a service or action, at least one second interface information corresponding to the i-th first interface information can be registered as at least one MCP tool, where i is a positive integer and i is any value between 1 and m.

[0169] If the message mechanism corresponding to the i-th first interface information among the m first interface information is a topic, at least one second interface information corresponding to the i-th first interface information can also be registered as at least one MCP resource.

[0170] After registering the second interface information corresponding to each of the m first interface information as an MCP tool or MCP resource, the p second interface information corresponds to k MCP functions. These k MCP functions include MCP tools and / or MCP resources, where k is a positive integer.

[0171] At least one robot's state information is acquired using ROS. This state information includes at least one of the following: robot battery level, temperature, motion pose, motion speed, task status, and environmental information. The state information corresponds to at least one robot's finite state machine (FSM). This state information is then registered as an MCP cue.

[0172] A Functional Model (FSM) is a computational model that represents the transitions between different states of a system. It consists of a set of states, inputs, outputs, and state transition rules. FSMs are widely used in digital circuits, computer science, and software development to model and control system behavior. Therefore, here we use an FSM to maintain the robot's state information, aggregating multi-source information from the Robotic Operating System (ROS) in real time, including the robot's hardware state (e.g., battery level, temperature), motion state (e.g., pose, velocity), task state (e.g., action feedback), and environmental information. Then, the discrete state values ​​are abstracted into high-level semantic states, including idle, executing, error, and waiting states.

[0173] The transition logic for various states in the FSM of the above robot is as follows: Figure 4 As shown. Upon startup, the robot is in idle state; while performing a task, it is in the "Executing" state; after the task is completed, it returns to idle state. If a fault occurs during execution, the status is "Error"; with manual intervention, the status is "Waiting for Confirmation". When a blacklisted task is received while idle, the status is "Waiting for Confirmation"; after user confirmation, the status is "Executing"; if the user rejects, it returns to idle state. When in a fault state, it returns to idle state after the fault is cleared.

[0174] MCP is a standardized middleware layer between ROS and LLM. ROS provides an operational interface through a messaging mechanism of services, actions, and topics, and reflects the robot's state through nodes. MCP includes three functions: tools, resources, and prompts. Therefore, this embodiment maps the concepts of ROS message interfaces and nodes to the functions of MCP. Specifically, it registers the second interface information corresponding to ROS service and action interfaces as tools in MCP, registers the second interface information corresponding to ROS topic interfaces as resources in MCP, and registers the state information of robot nodes as prompts in MCP to facilitate robot control.

[0175] The mapping logic between ROS's message interface, node concepts, and MCP functions is shown in Table 2 below:

[0176] ROS concept MCP function Mapping Logic Example service tool Request -> Input Parameters -> Response -> Return Result / spawn->spawn_turtle action tool It is divided into two tools: "Trigger Task" and "Query Results". / navigate->avigate_goal,navigate_result topic resource Topic Name -> URI Message Content -> Blob / scan->ros: / / robot / scan node prompt Node Status -> System Prompt amcl->LocalizationContext

[0177] Table 2

[0178] The mapping logic in Table 2 above will be explained in detail below:

[0179] 1. Service-to-tool mapping. Services use a request-response messaging mechanism, similar to RPC. Registering the service interface as a tool in an MCP allows the MCP to use the tool via function calls.

[0180] You can also convert ROS hierarchical namespaces (such as / turtle1 / teleport) into flat tool names (such as turtle1_teleport) to conform to the LLM function call convention.

[0181] The parameters of the service request are extracted as the input parameters schema of the tool, and the service response is defined as the result returned by the tool.

[0182] 2. Mapping from Action to Tool. Actions are suitable for long-cycle tasks. ROS's long-cycle tasks are broken down into two independent MCP tools: one for "Triggering the Task" and the other for "Querying Results." First, the "Triggering the Task" tool is called to start the task. Then, the "Querying Results" tool is called via polling or event listening, thus simulating the execution flow of executing the task and asynchronously retrieving the results within the LLM.

[0183] 3. Topic-to-Resource Mapping: ROS topics include pub and subscribe. ROS nodes can actively publish messages to share robot sensor data using the ROS topic mechanism, and can also subscribe to robot sensor data from ROS nodes, similarly providing a message interface using the ROS topic mechanism. ROS topics are mapped to resources in MCP, assigning a unique Uniform Resource Identifier (URI) to each topic, such as ros: / / hostname / topic_name, allowing MCP to access robot sensor data like a webpage. Message content is mapped to resources in Binary Large Object (Blob) format for easier processing in subsequent steps.

[0184] 4. Node-to-Prompt Mapping. ROS contains multiple robot nodes, each executing at least one specific robot operation as a process. Through the node-to-prompt mapping, the MCP Client can obtain robot status prompts through MCP topics and provide the user with LLM prompts about the current robot status in a contextual manner.

[0185] S309, Update the MCP Registry. The MCP Registry is a centralized public directory and open API used to manage and discover publicly accessible MCP functionalities. Updating the registry makes MCP functionalities registered with the second interface information available for use.

[0186] S310, Notify the MCP Client. The MCP Client can be notified via notifications / tools / list_changed to refresh the feature list, and to notify the MCP Client of the features registered with the second interface information, for use by both the LLM and MCP Clients.

[0187] S311, waiting for a call. After completing the above steps, you can wait for user instructions to execute subsequent operations.

[0188] Steps S301 to S311 described above implement the acquisition and mapping of ROS interface capabilities, including obtaining ROS interface information, parsing semantic information, and registering MCP functions. Steps S302 to S310 can be executed at device startup, periodically during device operation at a specified frequency, or, based on the ROS interface callback mechanism, to discover new interfaces in real time and then execute the above steps for those new interfaces. This application does not limit the execution method of steps S302 to S310.

[0189] Optionally, the functions of steps S301 to S311 above, namely the acquisition and mapping of ROS interface capabilities, can also be achieved through a preset mapping configuration file. Defining the mapping relationship between the ROS interface and MCP functions through a mapping configuration file (such as mapping.yaml) can speed up device startup, ensure absolute control over the mapping relationship, and enhance security, making it suitable for industrial robot scenarios with relatively fixed functions. This mapping configuration file can be manually written or exported from existing equipment; this application does not limit the method of generating the mapping configuration file.

[0190] The above process for obtaining and mapping ROS interface capabilities through the mapping configuration file is as follows: Figure 5As shown. After the device starts up, it loads the mapping configuration file, confirms the configuration is valid, parses the mapping configuration file, reads the tool definition and parameter mode, then binds the ROS interface, registers the MCP function, and the service is ready to wait for the MCP Client to call.

[0191] Steps S301 to S311 above realize the acquisition and mapping of ROS interface capabilities. Subsequent steps S312 to S320 include the specific process of user-robot interaction. These steps are described in detail below.

[0192] S312 responds to user operation commands.

[0193] S313, obtain status prompt information associated with the robot's status information through MCP prompts.

[0194] In step S308 above, the robot's FSM (Fundamental System Message) is established to maintain the robot's state, and the robot's state information is registered as a prompt for the MCP (Multi-Channel Programming). Here, through the MCP's prompting function, a structured system prompt generated based on the robot's FSM and state information can be obtained. The system prompt can contain the following information:

[0195] 1. Environmental snapshot, including the robot's current location, remaining battery power, and information on surrounding obstacles.

[0196] 2. Capability constraints: The set of tools that are allowed to be called in the current state. For example, "Map Reset" is prohibited in the "Navigating" state.

[0197] 3. Interaction suggestions to guide LLMs on how to respond to specific user inquiries.

[0198] In some implementations, steps S312 and S313 can be executed in the order described above. Alternatively, step S313 can be executed first to display the robot's status information to the user, prompting the user to input operation commands, and then step S312 can be executed to respond to the user's natural language operation commands before proceeding with subsequent operations. This application does not limit the execution order of the two steps described above.

[0199] S314 generates operation commands based on LLM, robot status prompts, and p second interface information.

[0200] In step S312 above, the user's operation instructions include a first natural language instruction. The LLM understands the ROS interface based on the second interface information, interprets the user's first natural language instruction, and selects q second interface information from p second interface information according to the robot's state information to generate a first operation command. The first operation command corresponds to q second interface information. After receiving the first operation command, the MCP Client sends the first operation command to the MCP Server, where q is a positive integer and q is less than or equal to p.

[0201] Optionally, in the preceding step S308, p second interface information has been registered as k MCP functions, including MCP tools and / or resources. Therefore, the first operation command can also be generated based on the LLM and the k MCP functions. Alternatively, the first operation command can also be generated based on status prompt information, the LLM, and the k MCP functions.

[0202] Understandably, the robot's status cues mentioned above enable the LLM to generate more accurate and appropriate operation commands. For example, if a user's natural language command is "make the robot move forward," without robot status cues, the LLM can only generate the operation command simply according to the user's instruction. However, by organizing the robot's status information, such as "current pose," "target area reachability," "currently navigating," and "battery level," into status cues, the LLM can generate more suitable operation commands based on these cues. Furthermore, when the robot is already in the "executing" state, the LLM will avoid generating operation commands that conflict with the robot's current actions based on the corresponding status cues; and when the robot is in the "waiting" state, the LLM will prioritize waiting for the user's risk confirmation rather than directly generating an operation command. Therefore, this method ensures that the LLM can generate more reasonable and accurate operation commands based on the robot's current state.

[0203] Optionally, the System Prompt generated in step S303 can be inserted at the very beginning of the user's dialogue history and sent to the LLM along with the user's instructions, so that the LLM can prioritize the robot's state and give the most reasonable response.

[0204] In some implementations, when the semantic information corresponding to the q second interface information includes the constraint conditions of the parameters, the LLM can also refer to the above parameter constraint conditions when generating operation commands, and perform a second layer of security filtering in the operation command generation stage, so that the generated operation commands avoid parameter errors.

[0205] S315 performs security filtering on operation commands.

[0206] This step includes a second layer of security filtering based on parameter constraints and a third layer of security filtering based on blacklists and whitelists. Specifically:

[0207] After receiving the first operation command, the MCP Server performs security filtering on the first operation command based on at least one of the following before executing subsequent operation steps:

[0208] Ensure that the first operation command is within the whitelist of the set second operation interface;

[0209] Once the first operation command is confirmed to be in the blacklist of the second operation interface, the user is prompted to confirm whether to execute the first operation command, and the user's confirmation is received.

[0210] Optionally, in step S312 above, the user's operation instruction further includes a second natural language instruction. In step S314 above, a second operation command is generated based on the second operation command, the robot's state information, the LLM, and p second interface information. In step S315, at least one of the following operations is performed on the second operation command:

[0211] If the second operation command is not in the whitelist of the second operation interface set above, the second operation command is discarded.

[0212] If the second operation command is in the blacklist of the second operation interface set above, the user is prompted to confirm whether to execute the second operation command. In response to the user's denial, the second operation command is discarded.

[0213] After the second operation command mentioned above is discarded, feedback can be given to the user, and the user's subsequent operation instructions can be awaited.

[0214] Understandably, the aforementioned pre-defined second operation interface whitelist can exclude high-risk interfaces of ROS, such as "system reset" or "formatting," thus limiting robot operations to the whitelist and making robot control safer.

[0215] The blacklist confirmation mechanism employs a Human-in-the-Loop (HITL) confirmation process for high-risk operations. When an operation command is on a pre-defined second operation interface blacklist, the execution flow is automatically suspended, and a "Pending Confirmation" signal is sent to the MCPClient. The MCP Client then forwards this signal to the user for confirmation, displaying a risk warning for the operation. Only after the user explicitly clicks "Confirm" on the interface will the authorization command be sent back to the MCP Server and subsequently to the ROS; otherwise, the command is discarded. For example, the second operation interface blacklist may include high-risk operations such as "Delete Map" and "High-Speed ​​Cruise." This mechanism effectively allows for manual intervention in high-risk operations, making robot operation safer.

[0216] Optionally, a second layer of security filtering can be performed before the operation command is executed, based on parameter constraints. That is, after receiving the first operation command, the MCP Server performs the following security filtering on the first operation command before executing subsequent operation steps:

[0217] Given that q second interface information corresponds to c parameters, and c parameters correspond to d constraints, determine that the value of the parameter corresponding to the first operation command satisfies d constraints. The d constraints are contained in the semantic information corresponding to the q second interface information, where c is a positive integer and d is a positive integer.

[0218] In some implementations, in step S312 above, the user's operation instruction further includes a third natural language instruction. In step S314 above, a third operation command is generated based on the third natural language instruction, the robot's state information, the LLM, and p second interface information. The third operation command corresponds to r second interface information among the p second interface information, where r is a positive integer and r is less than or equal to p. The following second-layer security filtering is performed on the third operation command:

[0219] Given r second interface information corresponding to e parameters, and e parameters corresponding to f constraints, if the value of the parameter corresponding to the third operation command does not satisfy the f constraints, the third operation command is discarded. The f constraints are contained in the semantic information corresponding to the r second interface information, where e is a positive integer and f is a positive integer.

[0220] After the third operation command mentioned above is discarded, the user can be given feedback of "parameter error" and wait for the user's subsequent operation instructions.

[0221] By utilizing the parameter constraints contained in the semantic information corresponding to the second interface information, rigorous type checks and numerical range verifications are performed on the parameters of the operation commands. For example, the linear velocity must not exceed 0.5 m / s. This prevents parameter out-of-bounds errors caused by LLM mathematical calculation errors or unit confusion, protecting the robot hardware from damage. Therefore, semantic information not only helps the LLM understand the ROS interface functions, but also allows for the safety verification of operation commands using its contained parameter constraints, making robot control safer.

[0222] In some implementations, the blacklist / whitelist-based security filtering can be divided into two parts: filtering on the whitelist and manual verification on the blacklist. Therefore, the three security checks based on whitelists, blacklists, and parameter constraints can use only one of them, or all three can be used in combination. One way to combine the three security checks is as follows... Figure 6 As shown, the process is as follows: When the MCP Server receives an operation command, it first performs a whitelist check. Commands not in the whitelist are discarded, and the user is returned an "Access Denied" message. Next, parameter validation is performed. Commands that do not meet the parameter constraints are also discarded, and the user is returned a "Parameter Error" message. Finally, a risk assessment is conducted based on the blacklist. For high-risk operations, manual confirmation from the user is required; otherwise, the command is discarded. Only operation commands that pass the above three security checks can continue to execute subsequent steps.

[0223] Optionally, when the above three security checks are combined according to Figure 6 When using the illustrated process, the execution order of the first two security checks can be adjusted. For example, parameter constraint validation can be performed first, followed by whitelist-based filtering. This application does not impose any restrictions on the execution order of these first two security checks.

[0224] It should be noted that the second operation interface whitelist in this step can be the same as the first operation interface whitelist in step S303 above, or different first and second operation interface whitelists can be set according to the different security requirements of the interface discovery phase and the command execution phase. Similarly, the second operation interface blacklist in this step can be the same as the first operation interface blacklist in step S303 above, or different first and second operation interface blacklists can be set according to the different security requirements of the interface discovery phase and the command execution phase.

[0225] In some implementations, the aforementioned security check strategy can be dynamically adjusted during device startup and operation. During device startup, the MCP Server can load security policies from a preset security configuration file, including a whitelist of second operation interfaces and a blacklist of second operation interfaces containing high-risk operation sets. During device operation, administrators can dynamically adjust the security mode through the MCP interface, for example, switching from "startup mode" to "running mode," or from "development mode" to "production mode." Different modes can correspond to different security policies to adapt to different security requirements in different environments. Optionally, the preset security configuration file may also include parameter constraints.

[0226] In step S303 above, the first layer of security filtering is performed on the first interface information during the interface capability discovery phase based on blacklists and whitelists, which can strictly limit which interfaces can be used. In steps S314 above and this step S315, the second layer of security filtering is performed during the operation command generation phase and before execution based on parameter constraints, which can avoid parameter errors. In this step, the third layer of security filtering is based on the whitelist filtering and blacklist manual confirmation mechanism, which can further limit the interfaces that are allowed to operate and can manually confirm high-risk operations. Through the above three-level security filtering mechanism, security control is achieved at each stage of robot control, making robot operation safer and risk controllable.

[0227] S316, based on ROS and MCP, instruct at least one robot to execute a first operation command and obtain a second return information.

[0228] In step S308 above, the second interface information is registered as an MCP tool and resource. Here, these MCP tools and resources can be used to obtain the second return information.

[0229] In this step, the MCP Server converts the first operation command into a binary message from ROS in real time, and performs the conversion of the operation result into the second return information.

[0230] When the MCP instructs ROS to execute operation commands on the robot, a control flow transformation is performed, converting the operation commands from JSON Schema to ROS's binary form. Taking the execution of `nav_set_goal` as an example, the control flow transformation includes the following steps:

[0231] Step 1, Parameter Reception: Receive the various fields of the operation command, including the parameter object in JSON Schema, such as {"x":1.0,"y":2.0}.

[0232] Step 2, Dynamic Serialization: Based on the second interface information, fill each field of the above operation command into the ROS message object one by one.

[0233] Step 3, Binary Packaging: For ROS2, the serialization interface of the ROS middleware (RMW) is called to convert the message object into a binary stream of Common Data Representation (CDR) required by the DDS protocol. For ROS1, the corresponding message serialization method for ROS1 is used to convert the message object into a binary stream supported by the ROS1 interface.

[0234] Step 4, Command Issuance: Send the binary stream to the ROS node through the ROS Publisher or ROS ServiceClient object.

[0235] When a ROS node performs the above operations and publishes the results, the MCP Server performs the conversion of the ROS binary data stream into the second returned information. Taking the publication of sensor data / scan data by a ROS node as an example, the data stream conversion includes the following steps:

[0236] Step 1, Binary Capture: The MCP Server subscribes to the corresponding topic and receives the raw binary stream.

[0237] Step 2, Dynamic Deserialization: Use the AST structure to restore the binary stream into a Python / C++ message object.

[0238] Through the aforementioned control flow conversion and data flow conversion mechanisms, the generated first operation command can be accurately executed by the robot, and a second return information can be obtained.

[0239] S317 performs rate limiting, compression, and downsampling on the second returned information to generate a third returned information that conforms to MCP. Specifically, this includes the following operations:

[0240] If the frequency corresponding to the second returned information is greater than or equal to the set frequency threshold, the second returned information is rate-limited based on the token bucket algorithm to obtain the fourth returned information, and the third returned information is generated based on the fourth returned information.

[0241] If the second return information contains image and / or video data, the image and / or video data in the second return information is compressed to obtain the fifth return information, and the third return information is generated based on the fifth return information.

[0242] If the second return information contains point cloud data, voxelization downsampling is performed on the point cloud data in the second return information to obtain the sixth return information. Based on the sixth return information, the third return information is generated.

[0243] If the second return information contains environmental topology data, grayscale mapping is performed on the environmental topology data in the second return information to obtain the seventh return information. Based on the seventh return information, the third return information is generated.

[0244] The following is a detailed explanation of how to operate on the above types of data:

[0245] 1. High-Frequency Data Processing. In some scenarios, robot sensors, such as Inertial Measurement Units (IMUs), typically transmit data at frequencies above 100Hz. Directly forwarding such high-frequency data would instantly overwhelm network bandwidth and other data processing modules. Therefore, when the frequency of the second return message is greater than or equal to a set frequency threshold, the MCP Server sets up a token bucket rate limiter before converting the return data into the third return message. Tokens are added to the bucket at a set rate, with one token consumed for each forwarded message, thus limiting the flow of high-frequency data.

[0246] Optionally, smooth data stream shaping can be achieved in three ways: First, dynamically adjust the bucket capacity at runtime; second, adjust the frequency threshold to allow for short-term bursts of traffic, such as dense data returned when a collision is detected; and third, limit the long-term average transmission rate to a safe threshold (e.g., 10Hz). Using these methods allows for more reasonable smoothing and rate limiting of high-frequency data.

[0247] 2. Visual data processing: For camera images, a lossy compression strategy developed by the Joint Photographic Experts Group (JPEG) is adopted to generate JPEG format image data. This significantly reduces bandwidth usage while ensuring that the image is recognizable by the human eye, making it suitable for real-time monitoring scenarios.

[0248] For video data returned from monitoring screens, JPEG lossy compression is enabled to generate JPEG format image data. Simultaneously, features are extracted from visual localization and mapping (SLAM) data and converted into lossless compressed Portable Network Graphics (PNG) format image data.

[0249] Optionally, the JPEG format image data can be converted to Base64 encoding format for easier processing in subsequent steps.

[0250] 3. Point Cloud Data Processing: For LiDAR point clouds, voxelization downsampling is employed. The continuous spatial point cloud is discretized into a sparse voxel grid, retaining only the grid centroids, thereby reducing the data volume by an order of magnitude while preserving the geometric features of the environment.

[0251] Optionally, for point cloud data, the voxel grid size can be dynamically adjusted based on network bandwidth usage. Increase the voxel grid size when bandwidth is limited, and decrease it when bandwidth is sufficient, ensuring both point cloud accuracy and data flow as much as possible.

[0252] 4. Processing of environmental topology data: Taking the occupied grid map in the environmental topology data as an example, a grayscale mapping strategy can be used to map the occupancy probability value of the map to the grayscale value of the image pixels, generate a PNG image, which can be directly used as the base map for display to users.

[0253] Optionally, the returned information after processing the point cloud data and environmental topology data can be converted into Blob format data for easier processing in subsequent steps.

[0254] In addition, status data, such as battery level, is directly converted into text data in JSON string format.

[0255] The processed data is encapsulated into an MCP resource object, i.e., the third returned information, and marked with a standard Multipurpose Internet Mail Extensions (MIME) type, such as image / jpeg, and pushed to the MCP Client via a JSON RPC notification.

[0256] S318, MCP transmission.

[0257] Upon receiving an MCP resource update notification, the MCP Client proactively retrieves the latest resource content from the MCP Server, i.e., the third-party response information. It then distributes the data to different visualization processing components based on the MIME type.

[0258] S319, visualize the third returned information to obtain the first returned information.

[0259] The third return information obtained from processing visual data, point cloud data, and environmental topology data in step S317 above is restored into a visualized image using Web Graphics Library (WebGL) or Canvas technology, which is the first return information.

[0260] For the text-type status data in step S317 above, the visualization process is performed directly to display the corresponding text information to the user.

[0261] S320 displays the first return information obtained in step S319 to the user and proceeds to the next round of interaction.

[0262] In response to the user's fourth natural language instruction, a fourth operation command is generated based on the LLM, the aforementioned first return information, and p second interface information.

[0263] Based on ROS and MCP, at least one robot is instructed to execute a fourth operation command, and an eighth return message is obtained. This eighth return message is used to display the operation result corresponding to the fourth natural language instruction.

[0264] By using the visualized operation results as the basis for the next round of interaction, a complete and sustainable interaction chain can be formed, thereby enabling precise multi-round control of the robot.

[0265] In some implementations, embodiments of this application may also employ, such as Figure 7 The optimization process shown is optimized. Figure 7 The optimization processes for various ROS data sources, such as rate limiting, compression, and downsampling, are described in step S317 above. The optimized data transmitted over the network to the MCP Client is described in step S318 above. The following describes the caching and on-demand subscription content.

[0266] In step S315 above, the second interface information is needed to check the constraint conditions of the parameters. In step S316 above, the second interface information is also needed during the dynamic serialization operation. Therefore, the second interface information can be stored in the device's internal memory to improve the efficiency of command security checks and interface calls, reducing the time taken for a single call to the second interface information to the microsecond level.

[0267] During robot control, ROS nodes generate topic data in real time. The MCP Client only subscribes to the MCP Server's resources and ROS topic messages when the user opens an operation result window, such as a status dashboard or video window. At this time, the MCP Server retrieves the corresponding topic data, processes it, and returns it to the MCP Client for user display. When the user closes the window or switches pages, the MCP Client unsubscribes from the MCP Server's resources and subsequently cancels its ROS topic subscription, thus freeing up computing resources.

[0268] The following example illustrates the robot control method proposed in this application, using the control of robot navigation and data return as an example. This embodiment demonstrates how to use the above method to achieve natural language control, status monitoring, and visual feedback for a mobile robot.

[0269] The hardware platform for this embodiment is as follows:

[0270] Mobile robot chassis: TurtleBot3 Waffle Pi.

[0271] Onboard computer: Raspberry Pi 4B, 4 gigabytes (GB) of random access memory (RAM).

[0272] Sensors: 360-degree LiDAR (model: LDS-01), RGB color mode camera (model: Raspberry Pi Camera V2).

[0273] User terminal: MacBook Pro (M1 chip), running Chrome browser.

[0274] The software environment for this embodiment is as follows:

[0275] Operating system: Ubuntu 22.04 LTS (Jammy Jellyfish).

[0276] Robot middleware: ROS2 Humble Hawksbill.

[0277] MCP Server: Developed based on Python 3.10, integrating rclpy and mcp-sdk.

[0278] MCP Client: A web application developed based on Vue and TypeScript.

[0279] LLM: Claude 3.5 Sonnet accessed via API or Llama 3 deployed locally.

[0280] The steps in this embodiment are as follows:

[0281] Step 1: System startup and capability discovery.

[0282] Start the robot node: Run the ROS startup script on the Raspberry Pi to load the chassis driver, LiDAR driver, and camera node.

[0283] Start the MCP service: Run the MCP Server main program in the terminal and specify the communication method as standard input / output (Stdio).

[0284] Interface Acquisition: The MCP Server initializes the ROS connection and scans the current computation graph, discovering the following key interfaces:

[0285] Topic: / scan (LiDAR), / camera / image_raw (image), / odom (odometer).

[0286] Service: / spawn, / reset.

[0287] Action: / navigate_to_pose (navigation action).

[0288] Semantic mapping: Automatically generates corresponding MCP Tool and Resource functions. For example, / navigate_to_pose is mapped to the tool navigate_to_pose, with parameters including the target coordinates pose.

[0289] Step 2: Establish connection and context injection.

[0290] Users open the Chrome browser on a MacBook Pro and connect to the MCP Server via WebSocket.

[0291] The MCP Client calls list_tools and list_resources to retrieve the list of capabilities.

[0292] MCP Client subscribes to / battery_state and / robot_status.

[0293] MCP Client injects the initial System Prompt into the LLM:

[0294] "You are a robot assistant. The robot's battery is currently at 100% and it is in idle mode. Available tools include navigation and taking photos."

[0295] Step 3: Execution of natural language instructions.

[0296] The user inputs a natural language command: "Please move to the kitchen and take a picture and send it to me."

[0297] LLM analyzes the user's semantics and identifies two intentions: navigation and taking a photo.

[0298] Tool Call 1 (Navigation):

[0299] LLM call generation: call_tool("navigate_to_pose",{"x":3.5,"y":2.0,"theta":0.0}).

[0300] The security module of MCP Server verifies the parameter range (x, y are within the map boundary) and passes the verification.

[0301] The MCP Server converts the request into a ROS Action Goal and sends it to the / navigate_to_pose node.

[0302] Tool Call2 (Take a Photo):

[0303] LLM generation call: call_tool("get_camera_image",{}).

[0304] The MCP Server subscribes to / camera / image_raw and retrieves the latest frame image.

[0305] Step 4: Multimodal feedback and data display.

[0306] Navigation process echo:

[0307] The robot begins to move, and the / odom topic continues to post pose updates.

[0308] MCP Server pushes resource update notifications at a frequency of 5Hz (after rate limiting).

[0309] MCP Client updates the location of robot icons on the map in real time.

[0310] Image result feedback:

[0311] The MCP Server converts the acquired image into JPEG format and encodes it as a Base64 string.

[0312] The result of the get_camera_image tool is returned to the LLM.

[0313] LLM generated a reply: "Arrived at the kitchen, here is the current photo." along with an image resource link.

[0314] MCP Client renders content in Lightweight Markdown language and directly displays the captured on-site photos.

[0315] Using the above method, LLM accurately understands user commands, and MCP controls the robot's navigation and returns data through ROS, realizing natural language control, status monitoring, and visual feedback for a mobile robot.

[0316] This application also provides a robot control system, which includes a processing module that can implement the various processes of the above-described method embodiments and achieve the same technical effects. Specifically, the processing module of this system may include a user interaction layer, an intelligent interaction layer, a protocol bridging layer, and a robot execution layer.

[0317] The robot control system provided in this application embodiment can deploy each of the above layers on different electronic devices; alternatively, the user interaction layer and intelligent interaction layer can be deployed on one electronic device, and the protocol bridging layer and robot execution layer can be deployed on another electronic device; the robot execution layer can also be distributed and deployed on different electronic devices for robots with different distributions. This application does not limit the deployment method of the above layers.

[0318] The following is combined with Figure 8 and Figure 9 The robot control system provided in the embodiments of this application will be described in detail.

[0319] Figure 8 This paper illustrates the modular architecture of the robot control system provided in the embodiments of this application, as well as a schematic diagram of the data flow of each module in executing the various processes in the embodiments of this application. The system is divided into four layers from top to bottom: user interaction layer, intelligent interaction layer, protocol bridging layer, and robot execution layer. The functions of each layer and the modules it contains are described below.

[0320] 1. The User Interaction Layer is the direct interface between the system and the user, responsible for the input of natural language commands and the visualization of multimodal information. This layer includes the following modules:

[0321] Natural language interaction interface: Provides a chat window that supports text or voice command input.

[0322] Status monitoring dashboard: Displays the robot's status information in real time through visual charts, such as battery level, speed, position coordinates, and environment.

[0323] Multimodal rendering window: Utilizes WebGL or Canvas technology to render non-textual information returned by the robot, including real-time video streams (Motion Joint Photographic Experts Group, MJPEG), point clouds, and raster maps.

[0324] 2. Intelligent Interaction Layer: This layer is responsible for intent understanding, context management, and protocol adaptation. This layer includes the following modules:

[0325] MCP Client Core: Executes the functions of the MCP Client, including maintaining the connection with the MCP Server and managing the lifecycle of MCP tools and resources.

[0326] Context Manager: Acquires the robot's operating status in real time, dynamically constructs system prompts, and ensures that the LLM can perceive the current physical state of the robot and its environment.

[0327] LLM Adapter: Encapsulates different large language models (such as GPT-4, Claude, Llama) and converts natural language into standardized tool call requests.

[0328] Large Language Model (LLM): Provides LLM functionality through an LLM adapter. This module can be deployed on an electronic device as part of the intelligent interaction layer, or it can be deployed independently on other electronic devices as a standalone module that interacts with the MCP client core in the intelligent interaction layer through the LLM adapter.

[0329] 3. Protocol Bridge Layer: This layer includes the MCP Server and is responsible for ROS interface semantic self-discovery, security management, and optimization. This layer contains the following modules:

[0330] MCP server core: Executes the functions of the MCP Server.

[0331] ROS Introspection Module: Utilizes the ROS interface to dynamically scan the node topology and interface definitions in the ROS computation graph.

[0332] Semantic Mapping Module: Converts ROS's Interface Definition Language (IDL) into MCP's JSON Schema and injects semantic information.

[0333] Data conversion module: Enables efficient conversion between ROS binary messages and MCP text and / or Base64 formats.

[0334] Security control module: Performs whitelist filtering and parameter constraint verification to intercept dangerous commands.

[0335] Traffic shaping / limiter: Employs the token bucket algorithm to downsample high-frequency sensor data to prevent network congestion.

[0336] Subscription Manager: Manages subscriptions to and cancellations for ROS topics.

[0337] Metadata caching: Responsible for converting ROS interface information into JSON Schema conforming to MCP and storing it in the internal storage of electronic devices.

[0338] 4. Robot Execution Layer, which includes the following modules:

[0339] ROS Master / DDS Discovery: For ROS1, it is based on the ROS Master API; for ROS2, it is based on the DDS Discovery protocol. It is responsible for node management and interface discovery.

[0340] Functional node group: including navigation control node (MoveBase / Nav2), sensor drive node (Camera / Lidar) and actuator control node, etc., among which the actuator control node includes the chassis drive for operating the robot.

[0341] The communication and data interaction mechanisms between the above layers are as follows:

[0342] 1. User Interaction Layer and Intelligent Interaction Layer: Employing either the Document Object Model (DOM) event mechanism or local WebSocket. The user interaction layer triggers command events, which the intelligent interaction layer listens for and processes. The intelligent interaction layer drives the rendering of the user interface (UI) of the user interaction layer through state update events.

[0343] 2. Intelligent Interaction Layer and Protocol Bridging Layer: Adopts MCP protocol JSON-RPC2.0.

[0344] Transmission Channel: When the intelligent interaction layer and the protocol bridging layer are deployed on the same electronic device, a local process pipe that supports Stdio can be used. When the intelligent interaction layer and the protocol bridging layer are deployed on different electronic devices, server-sent events (SSE) can be used.

[0345] Optionally, when the intelligent interaction layer and the protocol bridging layer are deployed on different electronic devices, the HTTP REST API or WebSocket protocol can also be used to realize data communication between the intelligent interaction layer and the protocol bridging layer.

[0346] Control flow: The intelligent interaction layer sends a Tool Call request, and the protocol bridge layer returns the execution result.

[0347] Data flow: The protocol bridging layer actively pushes resource message updates, such as notifications / resources / updated, while the intelligent interaction layer obtains the latest data by subscribing to resources, such as resources / read.

[0348] 3. Protocol Bridging Layer and Robot Execution Layer: For ROS1, communication is achieved using ROSTCP (based on Transmission Control Protocol, TCP) and / or ROSUDP (based on User Datagram Protocol, UDP). For ROS2, middleware protocols such as DDS are used to implement real-time publish / subscribe communication.

[0349] like Figure 9 As shown, when LLM is deployed alone, the interaction steps with other modules in the intelligent interaction layer (hereinafter referred to as the intelligent interaction layer), the user interaction layer, the protocol bridging layer, and the robot execution layer to execute the various processes of the above method embodiments are as follows:

[0350] Step 1: The protocol bridging layer queries the robot execution layer to obtain the first interface information of ROS.

[0351] Step 2: The protocol bridging layer parses the first interface information to obtain the second interface information containing semantic information.

[0352] Step 3: The protocol bridging layer registers the second interface information as a function of MCP, and then notifies the intelligent interaction layer.

[0353] Step 4: The protocol bridging layer obtains robot sensor status data from the robot execution layer through querying or subscribing, and maintains the robot's status updates.

[0354] Step 5: The user interaction layer responds to the natural language instructions input by the user and sends the instructions to the intelligent interaction layer.

[0355] Step 6: The intelligent interaction layer queries the protocol bridging layer to obtain the robot status, and sends the user instructions, robot status, and second interface information to the large language model for analysis to obtain the operation commands.

[0356] Step 7: The intelligent interaction layer sends the operation command to the protocol bridging layer, which performs security filtering on the operation command.

[0357] Step 8: The protocol bridging layer instructs the robot execution layer to execute the filtered operation command and obtain the returned result.

[0358] Step 9: The protocol bridging layer performs rate limiting, compression, encoding, and downsampling on the returned result and sends it to the intelligent interaction layer. The intelligent interaction layer decodes the encoded data and then sends the returned data to the user interaction layer.

[0359] Step 10: The user interaction layer will process the returned data for visualization and show the user the operation results.

[0360] Step 11: Based on the visualization results from Step 10, proceed to the next round of user-robot interaction.

[0361] This application also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor. The memory is used to store computer program code, which includes computer instructions. One or more processors call the computer instructions to cause the electronic device to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0362] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0363] The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. In some examples, the readable storage medium may be a non-transient readable storage medium. This application also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments, achieving the same technical effects. To avoid repetition, it will not be described again here.

[0364] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here. It should be understood that the chip mentioned in this application embodiment can also be called a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0365] In summary, the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0366] In the embodiments of this application, the term "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, X and / or Y can represent: X alone, X and Y simultaneously, or Y alone, where X and Y can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0367] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. Furthermore, the terms "comprising" and "having" in the embodiments, claims, and drawings of this application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules and may also include steps or modules not listed.

[0368] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

Claims

1. A control method of a robot characterized by, A control system applied to a robot, the method comprising: Obtain m first interface information of the robot operating system ROS, where m is a positive integer, and each first interface information includes the interface name and interface type; Based on the interface type of the i-th first interface information among the m first interface information, obtain the interface definition file corresponding to the i-th first interface information, where i is a positive integer and i is any value between 1 and m; Based on an abstract syntax tree (AST), the interface definition file is recursively parsed to obtain the annotation information corresponding to the i-th first interface information and parameters, is a natural number, and each parameter includes a parameter name and a data type. Based on natural language processing (NLP) technology, semantic information corresponding to the i-th first interface information is acquired from the annotation information, and the semantic information includes interface function description corresponding to the i-th first interface information. In a case where the one parameter is at least one parameter, the semantic information further includes at least one of parameter function description, physical meaning, unit, constraint condition, and default value corresponding to the at least one parameter. based on the interface name of the ith first interface information, and the semantic information and the m parameters corresponding to the ith first interface information, generate at least one second interface information conforming to a model context protocol (MCP), the at least one second interface information corresponding to the interface name, the semantic information and the m parameters corresponding to the ith first interface information The m first interface information corresponds to p second interface information, p is a positive integer, p is greater than or equal to m, is a natural number;​​ In response to the user's first natural language instruction, a first operation command is generated based on the Large Language Model (LLM) and the p second interface information. The first operation command corresponds to q second interface information among the p second interface information, where q is a positive integer and q is less than or equal to p. Based on the ROS and the MCP, at least one robot is instructed to execute the first operation command to obtain first return information, which is used to display the operation result corresponding to the first natural language instruction.

2. The method according to claim 1, characterized in that, The acquisition of m first interface information of ROS includes: Obtain n first interface information of the ROS, where n is a positive integer and n is greater than or equal to m; The m first interface information are obtained by performing security filtering on the n first interface information based on at least one of the following: Security filtering is performed on the n first interface information based on the set first operation interface whitelist, and the resulting m first interface information are all in the first operation interface whitelist; Based on the established first operation interface blacklist, the n first interface information is security filtered, and the resulting m first interface information are all not in the first operation interface blacklist.

3. The method according to claim 1, characterized in that, The first operation command, generated in response to the user's first natural language instruction, is based on the Large Language Model (LLM) and the p second interface information, including: If the message mechanism corresponding to the i-th first interface information is a service or an action, then at least one second interface information corresponding to the i-th first interface information is registered as at least one MCP tool. And / or, If the message mechanism corresponding to the i-th first interface information is a topic, then at least one second interface information corresponding to the i-th first interface information is registered as at least one MCP resource; Wherein, the p second interface information corresponds to k MCP functions, and the k MCP functions include MCP tools and / or MCP resources, where k is a positive integer; In response to the first natural language instruction, the first operation command is generated based on the LLM and the k MCP functions; Based on the ROS and the MCP, the process of instructing at least one robot to execute the first operation command and obtaining first return information includes: Based on the ROS and the MCP, the at least one robot is instructed to execute the first operation command; The first returned information is obtained based on at least one of the k MCP functions.

4. The method according to claim 1, characterized in that, The method further includes: The state information of the at least one robot is obtained based on the ROS. The state information includes at least one of the robot's battery level, temperature, motion pose, motion speed, task status, and environmental information, and the state information corresponds to the finite state machine (FSM) of the at least one robot. Register the status information as an MCP prompt; The first operation command, generated in response to the user's first natural language instruction, is based on the LLM and the p second interface information, including: The status prompt information associated with the status information is obtained through the MCP prompt; In response to the first natural language instruction, the first operation command is generated based on the status prompt information, the LLM, and the p second interface information.

5. The method according to claim 1, characterized in that: Before instructing at least one robot to execute the first operation command based on the ROS and the MCP, the method further includes at least one of the following: Determine that the first operation command is within the set second operation interface whitelist; Once the first operation command is confirmed to be in the set second operation interface blacklist, the user is prompted to confirm whether to execute the first operation command, and the user's confirmation operation is received. And / or, The method further includes: In response to the user's second natural language instruction, a second operation command is generated based on the LLM and the p second interface information; If the second operation command is not in the whitelist of the second operation interface, discard the second operation command; or... If the second operation command is in the blacklist of the second operation interface, the user is prompted to confirm whether to execute the second operation command. In response to the user's denial, the second operation command is discarded.

6. The method according to claim 1, characterized in that: Before instructing at least one robot to execute the first operation command based on the ROS and the MCP, the method further includes: When the q second interface information corresponds to c parameters and the c parameters correspond to d constraints, it is determined that the value of the parameter corresponding to the first operation command satisfies the d constraints, and the d constraints are contained in the semantic information corresponding to the q second interface information, where c is a positive integer and d is a positive integer; And / or, The method further includes: In response to the user's third natural language instruction, a third operation command is generated based on the LLM and the p second interface information. The third operation command corresponds to r second interface information among the p second interface information, where r is a positive integer and r is less than or equal to p. When the r second interface information corresponds to e parameters and the e parameters correspond to f constraints, if the value of the parameter corresponding to the third operation command does not satisfy the f constraints, the third operation command is discarded. The f constraints are contained in the semantic information corresponding to the r second interface information, where e is a positive integer and f is a positive integer.

7. The method according to any one of claims 1 to 6, characterized in that, Based on the ROS and the MCP, the process of instructing at least one robot to execute the first operation command and obtaining first return information includes: Based on the ROS and the MCP, the at least one robot is instructed to execute the first operation command to obtain the second return information; Based on the second returned information, a third returned information conforming to the MCP is generated; The third returned information is visualized to obtain the first returned information.

8. The method according to claim 7, characterized in that, The generation of third return information conforming to the MCP based on the second return information includes at least one of the following: If the frequency corresponding to the second returned information is greater than or equal to the set frequency threshold, the second returned information is rate-limited based on the token bucket algorithm to obtain the fourth returned information, and the third returned information is generated based on the fourth returned information. If the second return information contains image and / or video data, the image and / or video data in the second return information is compressed to obtain the fifth return information, and the third return information is generated based on the fifth return information; If the second returned information contains point cloud data, voxel downsampling processing is performed on the point cloud data in the second returned information to obtain the sixth returned information, and the third returned information is generated based on the sixth returned information. If the second returned information contains environmental topology data, the environmental topology data in the second returned information is subjected to grayscale mapping processing to obtain the seventh returned information, and the third returned information is generated based on the seventh returned information.

9. The method according to any one of claims 1 to 6, characterized in that, After instructing at least one robot to execute the first operation command based on the ROS and the MCP, and obtaining the first return information, the method further includes: In response to the user's fourth natural language instruction, a fourth operation command is generated based on the LLM, the first return information, and the p second interface information; Based on the ROS and the MCP, the at least one robot is instructed to execute the fourth operation command to obtain the eighth return information, which is used to display the operation result corresponding to the fourth natural language instruction.

10. A control system for a robot, characterized in that, It includes a processing module for implementing the method of any one of claims 1 to 9.

Images