Intelligent execution method, device and equipment of tool body, storage medium and program product

Abstract

The embodiment of the invention provides an intelligent implementation method, device and equipment for a tool body, a storage medium and a program product. The method is applied to a client, the client is deployed on the edge side of a robot, a plurality of services are registered in the client, the services are called functions, and the plurality of services communicate based on a model context (MCP) protocol; the method comprises the following steps: receiving an instruction input by a user; sending the instruction and the available service information to a large language model to enable the large language model to generate service response information; service response information is received, to-be-called services in the service response information are called through a server side, a service execution result fed back by the server side is received, and the server side is deployed on the edge side of the robot; and sending the service execution result to the large language model to enable the large language model to generate a natural language response based on the service execution result, receiving the natural language response and displaying the natural language response to the user. According to the invention, the compatibility between services is improved, and the intelligent execution efficiency of the user is improved.

Description

Technical Field

[0001] The present application relates to embodied intelligence technology, and in particular to an embodied intelligence execution method, apparatus, device, storage medium, and program product. Background Art

[0002] Embodied intelligence refers to the ability to achieve intelligent behavior through the interaction between the body and the environment. Robots with embodied intelligence have been widely used in various fields, such as logistics robots for sorting objects and service robots for hospitality.

[0003] At present, in the robot sorting scenario, the existing technology can be based on the "vision-motion" integrated system of deep learning, use convolutional neural networks to process image information, and combine reinforcement learning to optimize the grasping strategy of the robot arm.

[0004] However, in the existing technology, services that perform different tasks, such as vision-related services, motion-related services, and robot control-related services, have low compatibility and low collaborative efficiency, which in turn leads to low execution efficiency of embodied intelligence. Summary of the Invention

[0005] The present application provides an embodied intelligent execution method, device, equipment, storage medium and program product to improve the compatibility between different services, thereby improving the collaborative efficiency between different services and further improving the efficiency of intelligent execution.

[0006] In a first aspect, the present application provides a method for executing an intelligent body, which is applied to a client, wherein the client is deployed on the edge side of a robot, and the client registers multiple services, wherein the services are called functions, and the multiple services communicate based on the model context MCP protocol; the method includes:

[0007] Receive user input instructions;

[0008] Sending the instruction and available service information to a large language model so that the large language model generates service response information, wherein the available service information includes a plurality of services that can be called, and the service response information includes the service to be called;

[0009] Receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, where the server is deployed on the edge side of the robot;

[0010] The service execution result is sent to the large language model so that the large language model generates a natural language response based on the service execution result, and the natural language response is received and displayed to the user.

[0011] In one possible implementation, the instruction includes multiple subtasks, where the multiple subtasks are obtained by decomposing the instruction by the large language model, and each subtask has corresponding service response information; and receiving the service response information sent by the large language model includes:

[0012] After displaying the natural language response corresponding to the current subtask to the user, service response information corresponding to the next subtask sent by the large language model is received.

[0013] In one possible implementation, the method further includes:

[0014] After the client is started, it sends a connection request to the server and receives confirmation of the connection from the server;

[0015] Sending an available service information request to the server, and receiving available service information fed back by the server.

[0016] In a second aspect, the present application provides an embodied intelligent execution method, which is applied to a large language model, wherein the large language model is deployed on a server, and the method includes:

[0017] Receiving a user input instruction and available service information sent by a client, wherein the client has multiple services registered, the services being called functions, and the multiple services communicating based on the Model Context MCP protocol, and the available service information including multiple services that can be called;

[0018] Generate service response information according to the instruction and the available service information, wherein the service response information includes the service to be called;

[0019] Sending the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, where the server is deployed on the edge side of the robot;

[0020] Receive the service execution result fed back by the client, generate a natural language response according to the service execution result, and send the natural language response to the client.

[0021] In a possible implementation, generating service response information according to the instruction and the available service information includes:

[0022] The instruction is decomposed into tasks to obtain a plurality of subtasks, and a service corresponding to each subtask is determined according to the available service information to generate service response information corresponding to each subtask.

[0023] In a possible implementation, sending the service response information to the client includes:

[0024] After sending the natural language response corresponding to the current subtask to the client, service response information corresponding to the next subtask is sent to the client.

[0025] In a possible implementation, the available service information further includes: service description information of each service; and determining the service corresponding to each subtask according to the available service information includes:

[0026] Determine the service corresponding to each subtask based on the prompt information and the service description information of each service;

[0027] The prompt information includes a plurality of preset tasks and one or more services to be called for each task.

[0028] In a third aspect, the present application provides an embodied intelligent execution device, which is applied to a client, wherein the client is deployed on the edge side of a robot, and wherein the client registers multiple services, wherein the services are called functions, and the multiple services communicate based on the model context MCP protocol; including:

[0029] A receiving module, configured to receive instructions input by a user;

[0030] a sending module, configured to send the instruction and available service information to the large language model, so that the large language model generates service response information, wherein the available service information includes a plurality of services that can be called, and the service response information includes the service to be called;

[0031] The receiving module is further configured to receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, wherein the server is deployed on the edge side of the robot;

[0032] The sending module is further configured to send the service execution result to the large language model, so that the large language model generates a natural language response based on the service execution result;

[0033] The receiving module is further configured to receive the natural language response and display it to the user.

[0034] In a fourth aspect, the present application provides an embodied intelligent execution device, applied to a large language model, wherein the large language model is deployed on a server; comprising:

[0035] a receiving module, configured to receive user input instructions and available service information sent by a client, wherein the client has multiple services registered, each of which is a called function, and the multiple services communicate based on the Model Context MCP protocol, and the available service information includes multiple services that can be called;

[0036] a processing module, configured to generate service response information according to the instruction and the available service information, wherein the service response information includes the service to be called;

[0037] A sending module, configured to send the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, wherein the server is deployed on the edge side of the robot;

[0038] The receiving module is further configured to receive the service execution result fed back by the client;

[0039] The processing module is further configured to generate a natural language response according to the service execution result, and send the natural language response to the client.

[0040] In a fifth aspect, the present application provides an embodied intelligent execution device, comprising: a memory, a processor;

[0041] The memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method described in any one of the first aspects or the method described in any one of the second aspects.

[0043] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any one of the first aspect or the method described in any one of the second aspect.

[0044] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of the first aspect or the method described in any one of the second aspect.

[0045] The present application provides an embodied intelligent execution method, apparatus, device, storage medium and program product, which is applied to a client, which is deployed on the edge side of a robot. The method includes: receiving instructions input by a user, and sending the instructions and available service information to a large language model. The large language model can generate service response information based on the instructions and available service information and send it to the client. The client can call the service to be called in the service response information through the server, and receive the service execution result fed back by the server. The service execution result is sent to the large language model, and the large language model generates a natural language response based on the service execution result and sends it to the client. The client displays the natural language response to the user. Since in this application, multiple services communicate based on the MCP protocol, the MCP protocol serves as a unified access layer, which improves the compatibility between different services, reduces the difficulty of adapting different services, and improves the collaborative efficiency, which is conducive to improving the efficiency of embodied intelligent execution. At the same time, it also effectively supports the functional expansion of embodied intelligent scenarios, breaks the platform dependency, and enables any service that supports the MCP protocol to be seamlessly connected, thereby improving scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of an application scenario of embodied intelligence provided by an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process of an embodied intelligent execution method provided in an embodiment of the present application Figure 1 ;

[0049] Figure 3 A schematic diagram of the framework of an embodied intelligent system based on the MCP protocol provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a process of an embodied intelligent execution method provided in an embodiment of the present application Figure 2 ;

[0051] Figure 5 An interactive flow chart of an embodied intelligent execution method provided in an embodiment of the present application;

[0052] Figure 6 A flow chart of a method for determining the location of a container provided in an embodiment of the present application;

[0053] Figure 7 A flow chart of a method for determining a driving path to a container location provided in an embodiment of the present application;

[0054] Figure 8 A flow chart of a method for moving to a container location provided in an embodiment of the present application;

[0055] Figure 9 A flow chart of a method for determining the position of a Coke bottle provided in an embodiment of the present application;

[0056] Figure 10 A flow chart of a method for grabbing a Coke bottle provided in an embodiment of the present application;

[0057] Figure 11 A flow chart of a method for determining the location of user A provided in an embodiment of the present application;

[0058] Figure 12 A flow chart of a method for determining a driving path to user A provided in an embodiment of the present application;

[0059] Figure 13 A flow chart of a method for moving to the location of user A provided in an embodiment of the present application;

[0060] Figure 14 A schematic diagram of the structure of an embodied intelligent execution device provided in an embodiment of the present application Figure 1 ;

[0061] Figure 15 A schematic diagram of the structure of an embodied intelligent execution device provided in an embodiment of the present application Figure 2 ;

[0062] Figure 16 A schematic diagram of the structure of an embodied intelligent execution device provided in an embodiment of the present application.

[0063] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] Figure 1 A schematic diagram of an application scenario of embodied intelligence provided in an embodiment of the present application is shown in FIG. Figure 1As shown, user B interacts with the robot and issues a voice command to the robot: "Go to the container and get a bottle of Coke and give it to user A." After receiving this command, the robot moves to the container, takes out the Coke, and then moves to user A's location to complete the command. Figure 1 The container and user A are not shown.

[0066] In robotic sorting scenarios, the technical background primarily involves the integration of machine vision, deep learning, and robotic arm control. A typical implementation typically involves the following steps: first, image data of the target object is captured using an industrial camera or 3D sensor. Second, the object is identified and located using traditional image processing algorithms, such as edge detection and contour extraction, or deep learning-based object detection and segmentation models, such as YOLO and Mask R-CNN. The recognition results are then converted into motion trajectory planning for the robotic arm, generating joint angle commands through inverse kinematics calculations. Finally, the robotic arm performs a grasping action to complete the sorting task.

[0067] One existing technology is a "vision-motion" integrated system based on deep learning. This solution uses convolutional neural networks (CNN) to process image information and combines reinforcement learning to optimize the grasping strategy of the robotic arm.

[0068] However, existing technologies lack compatibility and collaboration between services that perform different tasks, such as vision-related services, motion-related services, and robotic control-related services. This results in low efficiency in implementing embodied intelligence. Furthermore, existing technologies require large amounts of labeled data for model training, increasing deployment costs and time.

[0069] At the same time, the traditional manual hard coding method is inefficient and has poor adaptability to complex scenes, especially when objects are stacked or occluded, the recognition accuracy decreases.

[0070] To improve adaptability to complex scenarios, an automated solution for function calls in large intelligent models can be adopted. However, this solution creates an ecological barrier due to incompatible APIs (Application Programming Interfaces) between platforms. For example, the function call mechanisms of OpenAI and Google are not mutually exclusive, resulting in high adaptation costs and code refactoring risks for developers when switching large models. Furthermore, when adding new function calls, the original code requires extensive adaptation work, resulting in high development costs. Finally, the service ecosystem is severely fragmented, and there is a lack of unified standards for function expansion across different platforms. For embodied intelligent hardware carriers such as robotic arms, in order to realize different business scenarios, they need to be adapted to each scenario one by one, resulting in poor scalability and the need for customized function code development.

[0071] Based on this, the present application provides an embodied intelligent execution method, which sets up communication between multiple services based on the model context MCP protocol. The MCP protocol serves as a unified access layer, which improves the compatibility between different services, reduces the adaptation difficulty of different services, and improves the collaborative efficiency, thereby helping to improve the efficiency of embodied intelligent execution. At the same time, it also effectively supports the functional expansion of embodied intelligent scenarios, breaks the platform dependence, and enables any service that supports the MCP protocol to be seamlessly connected, thereby improving scalability.

[0072] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0073] Figure 2 A schematic diagram of a process of an embodied intelligent execution method provided in an embodiment of the present application Figure 1 , the method is applied to the client, the client is deployed on the edge side of the robot, the client registers multiple services, the services are called functions, and the multiple services communicate based on the model context MCP protocol, wherein the robot can be a wheeled, dual-arm robot, or a robot of other structures. The structure of the robot of this application is not limited. Figure 2 As shown, the method includes:

[0074] S201: Receive a command input by a user.

[0075] Before the user inputs an instruction, the client can be started. The instruction input by the user includes but is not limited to a text instruction, a voice instruction, etc. This application does not limit the type of the instruction input by the user.

[0076] In one implementation scenario, if the instruction is a voice instruction, the edge side of the robot perceives environmental information, collects the voice instruction issued by the user through an external microphone device, converts the voice instruction into a text instruction through a speech-to-text model, and inputs the text instruction into the client.

[0077] S202: Send the instruction and available service information to the large language model, so that the large language model generates service response information, wherein the available service information includes multiple services that can be called, and the service response information includes the service to be called.

[0078] Available service information includes multiple services and service descriptions for each service. These services include, but are not limited to, VLM (Visual Language Model) services, face recognition services, SLAM (Simultaneous Localization and Mapping) services, path planning services, body control services, and TTS (Text-to-Speech) services.

[0079] In addition to the above services, you can also customize other services related to embodied intelligence and add them to the edge client through registration to increase scalability.

[0080] In one implementation scenario, multiple services communicate based on the MCP protocol. The MCP protocol defines the format, type, and processing rules of messages, and is responsible for message transmission and interaction between services, ensuring accurate and efficient communication between different services.

[0081] Service description information includes but is not limited to the service's functional description, input parameters, and output format.

[0082] It should be noted that the above services can be deployed locally on the robot or on a cloud service. In this case, you only need to register the API interface corresponding to the service to the client, thereby reducing the local storage pressure of the robot.

[0083] In some embodiments, available service information can be obtained from a server. In one implementation scenario, after the client is started, it sends a connection request to the server and receives connection confirmation information from the server; it also sends an available service information request to the server and receives available service information from the server.

[0084] In one implementation scenario, when sending instructions and available service information to the large language model, they can be sent to the large language model through a function callback mechanism.

[0085] After receiving the instruction and available service information, the large language model generates a service response message. In one implementation scenario, the service response message includes multiple services required to complete the instruction input by the user, and the multiple services have a certain order.

[0086] In another implementation scenario, the large language model can decompose the instructions into tasks to obtain multiple subtasks, determine the service corresponding to each subtask based on the available service information, and generate service response information corresponding to each subtask. At this time, each service response information only includes one or more services required to complete the subtask.

[0087] S203: Receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, where the server is deployed on the edge side of the robot.

[0088] The edge side of the robot refers to the hardware, software, and computing resources deployed directly on the robot body or near-field devices.

[0089] In some embodiments, the server may provide the following three functions:

[0090] Resources, which can be read by clients, such as API responses or file contents.

[0091] 2. Tools, i.e. services, are functions that can be called by LLM and require user approval.

[0092] 3. Prompts are pre-written templates that help users complete specific tasks.

[0093] Figure 3 This is a schematic diagram of the framework of an embodied intelligent system based on the MCP protocol, provided in an embodiment of the present application. Resources can be local or remote, and each service can be associated with both local and remote resources. The modular design of multiple services not only reduces migration costs between services but also enables intensive integration of service resources through an open plug-in system, helping to alleviate the problem of a closed service ecosystem.

[0094] The above functions enable the server to provide rich contextual information and operational capabilities for embodied intelligence applications, thereby enhancing the practicality and flexibility of LLM. Therefore, when in use, the service is registered with the client and provided to the client for selection and use. For example, the body control service can directly call the control instructions of the robot body and control the robot to perform related actions through joint control instructions issued externally.

[0095] In one implementation scenario, when calling a service through a server, a service calling command may be sent to the server, where the service calling command carries the service to be called, so that the server calls the corresponding service.

[0096] In one implementation scenario, when the service response information includes multiple services, the client calls the multiple services in sequence through the server to complete the instructions input by the user or the corresponding subtasks.

[0097] The server may generate process data during the process of calling the service. In one implementation scenario, the process data can adopt a localized data processing mechanism, allowing users to customize the data interface and control the data transmission scope, thereby improving the level of automation while ensuring data sovereignty and reducing the risk of data circulation.

[0098] S204: Send the service execution result to the large language model, so that the large language model generates a natural language response based on the service execution result, receives the natural language response and displays it to the user.

[0099] In one implementation scenario, after receiving the natural language response, the client calls the TTS service through the server to generate voice feedback and play the voice feedback to the user, and / or generates text information corresponding to the natural language response and displays it to the user through the interface. This application does not limit the method of displaying the natural language response to the user.

[0100] In some embodiments, the instruction includes multiple subtasks, and the multiple subtasks are obtained by the large language model performing task decomposition on the instruction, and each subtask has corresponding service response information; receiving the service response information sent by the large language model includes:

[0101] After displaying the natural language response corresponding to the current subtask to the user, service response information corresponding to the next subtask sent by the large language model is received.

[0102] After receiving the service response information corresponding to the next subtask, the client calls the service specified in the service response information through the server. The service execution result received from the server is sent to the large language model, which generates the corresponding natural language response. After receiving the natural language response, it is displayed to the user.

[0103] Since users can get natural language responses for each subtask at this time, they can understand the completion status of the current instructions in real time, which is conducive to improving the user experience.

[0104] An embodiment of the present application provides an embodied intelligent execution method. After receiving the instruction input by the user, the client can send the instruction and available service information to the large language model, so that the large language model generates service response information, and the service response information contains the service to be called. After the client receives the service response information, it calls the service through the server and receives the service execution result fed back by the server. The service execution result is sent to the large language model, so that the large language model generates a corresponding natural language response based on the service execution result. The client receives the natural source response and displays it to the user. Since the multiple services registered in the client of the present application all communicate through the MCP protocol, the MCP protocol serves as a unified access layer, which improves the compatibility between different services, reduces the difficulty of adapting different services, and improves the collaborative efficiency, which is conducive to improving the efficiency of embodied intelligent execution. At the same time, it also effectively supports the functional expansion of embodied intelligent scenarios, breaks the platform dependency, and enables any service that supports the MCP protocol to be seamlessly connected, thereby improving scalability.

[0105] Figure 4 A schematic diagram of a process of an embodied intelligent execution method provided in an embodiment of the present application Figure 2 , applied to a large language model, which is deployed on a server, such as Figure 4 As shown, the method includes:

[0106] S401: Receive user input instructions and available service information sent by a client, wherein the client has multiple services registered, the services are called functions, and the multiple services communicate based on the model context MCP protocol, and the available service information includes multiple services that can be called.

[0107] Multiple services communicate based on the MCP protocol. The MCP protocol defines the format, type, and processing rules of messages, and is responsible for message delivery and interaction between services, ensuring accurate and efficient communication between different services.

[0108] The available service information includes multiple services and service description information of each service, wherein the service description information includes but is not limited to the function description, input parameters, and output format of the service.

[0109] S402: Generate service response information according to the instruction and the available service information, where the service response information includes the service to be called.

[0110] In one implementation scenario, generating service response information according to the instruction and the available service information includes:

[0111] The instruction is decomposed into tasks to obtain a plurality of subtasks, and a service corresponding to each subtask is determined according to the available service information to generate service response information corresponding to each subtask.

[0112] In some embodiments, the available service information further includes: service description information of each service; and determining the service corresponding to each subtask according to the available service information includes:

[0113] Determine the service corresponding to each subtask based on the prompt information and the service description information of each service;

[0114] The prompt information includes a plurality of preset tasks and one or more services to be called for each task.

[0115] Service description information includes but is not limited to the service's functional description, input parameters, and output format.

[0116] As can be seen from the above embodiments, the prompt information is provided by the server. In one implementation scenario, after receiving the available service information request sent by the client, the server feeds back the available service information to the client and also feeds back the prompt information to the client.

[0117] The prompt information may include multiple preset tasks and one or more services to be called for each task. For example, a face recognition service may be called to identify a target user.

[0118] In another implementation scenario, the prompt information may also include a role definition of the large language model and the tasks to be performed, for example, you are a robot assistant responsible for helping users complete specified tasks.

[0119] Determining the service corresponding to each subtask based on the prompt information is conducive to improving the efficiency and accuracy of determining the service.

[0120] In another implementation scenario, after determining the service corresponding to each subtask, the large language model can summarize the services corresponding to multiple subtasks and generate a service response information. The service response information includes multiple services required to complete the instruction, and the multiple services have a certain order.

[0121] In another implementation scenario, if the instruction is the same as one of the tasks in the prompt information, one or more services required to complete the instruction can be directly determined based on the prompt information, and service response information can be generated.

[0122] S403: Send the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, and the server is deployed on the edge side of the robot.

[0123] In one implementation scenario, when a client calls a service through a server, it can send a service call command to the server, which carries the service to be called. After the server completes the call, it can return the service execution result to the client.

[0124] S404: Receive the service execution result fed back by the client, generate a natural language response according to the service execution result, and send the natural language response to the client.

[0125] In one implementation scenario, if each subtask has corresponding service response information, sending the service response information to the client includes: after sending the natural language response corresponding to the current subtask to the client, sending the service response information corresponding to the next subtask to the client. The client executes the service required for the next subtask through the server and returns the service execution result to the large language model. The large language model generates a corresponding natural language response and returns it to the client. This process is repeated until the client receives the natural language response for the last subtask sent by the large language model, and the instruction is completed.

[0126] The embodied intelligence execution method provided by the embodiment of the present application is that the large language model receives the user input instructions and available service information sent by the client, and generates service response information according to the instructions and available service information. The service response information is sent to the client, so that the client calls the service to be called contained in the service response information through the server. The service execution result fed back by the client is received, a sub-natural language response is generated according to the service execution result, and the natural language response is sent to the client. Since in this application, since the multiple services registered in the client of this application all communicate through the MCP protocol, the MCP protocol serves as a unified access layer, which improves the compatibility between different services, reduces the adaptation difficulty of different services, and improves the collaborative efficiency, which is conducive to improving the efficiency of embodied intelligence execution, and also effectively supports the functional expansion of embodied intelligence scenarios, breaks the platform dependence, and enables any service that supports the MCP protocol to be seamlessly connected, thereby improving scalability. At the same time, based on the large language model and intelligent body related technologies, embodied intelligence service expansion and capability expansion are realized.

[0127] Based on the above examples, the following provides a specific example to illustrate the interaction between users, clients, servers, and a large language model. A client has been built on the edge side of the robot body, and various services that have been implemented, such as VLM service, face recognition service, SLAM service, path planning service, body control service, and TTS service, have been registered with the client.

[0128] Figure 5This is an interactive flow chart of an embodied intelligent execution method provided in an embodiment of the present application, refer to Figure 5 As shown, the method includes:

[0129] S501: The user starts the client.

[0130] S502: The client sends a connection request to the server.

[0131] S503: The server returns connection confirmation information to the client.

[0132] S504: The client sends a request for available service information to the server.

[0133] S505: The server returns available service information to the client.

[0134] S506: The user activates the robot interaction interface and inputs a voice command.

[0135] S507: The client receives a text instruction corresponding to the voice instruction. The text instruction is obtained by converting the voice instruction through a speech-to-text model of the robot body.

[0136] S508: The client sends the text instruction and available service information to the large language model.

[0137] S509: The large language model decomposes the text instruction into tasks to obtain multiple subtasks, and determines the service corresponding to each subtask to generate service response information corresponding to each subtask, and sends it to the client in sequence.

[0138] S510: After receiving the service response information, the client sends a service call command to the server.

[0139] S511: The server calls the service and returns the service execution result to the client.

[0140] S512: The client sends the service execution result to the large language model.

[0141] S513: The large language model generates a corresponding natural language response and sends it to the client.

[0142] S514: The client displays a natural language response.

[0143] The following describes in detail the process of embodied intelligence execution using a wheeled, dual-arm robot as the carrier and the instruction "go to the container and give the bottle of Coke to user A" as the scenario.

[0144] In one implementation scenario, the large language model decomposes the instruction "Go to the container and get the bottle of Coke to user A" into multiple subtasks, including: determining the container's location, determining the driving path to the container, moving to the container, determining the location of the Coke bottle, grabbing the Coke bottle, determining user A's location, determining the driving path to user A, and moving to user A's location. For each subtask, the interaction between the large language model, client, server, and user is as follows:

[0145] Figure 6 A flowchart of a method for determining the location of a container provided in an embodiment of the present application is provided. Figure 6 Shown, including:

[0146] S601: Based on the subtask of “determining the location of the container”, the large language model determines that the service to be called is the SLAM service, generates a first service response message, and sends it to the client.

[0147] S602: The client calls the SLAM service through the server to perceive the surrounding environment and determine the location of the container.

[0148] S603: The client returns the container location to the large language model.

[0149] S604: The large language model generates a first natural language response and returns it to the client.

[0150] S605: The client calls the TTS service through the server to return the first voice feedback.

[0151] S606: The client plays the first voice feedback to the user.

[0152] Figure 7 A flow chart of a method for determining a driving path to a container location provided in an embodiment of the present application, referring to Figure 7 Shown, including:

[0153] S701: Based on the subtask of “determining the driving path to the container location”, the large language model determines that the service to be called is the path planning service, generates a second service response message, and sends it to the client.

[0154] S702: The client calls the path planning service through the server to determine the path from the robot's current position to the container's position.

[0155] S703: The client returns the path from the robot's current location to the container location to the large language model.

[0156] S704: The large language model generates a second natural language response and returns it to the client.

[0157] S705: The client calls the TTS service through the server and returns the second voice feedback.

[0158] S706: The client plays the second voice feedback to the user.

[0159] Figure 8 A flow chart of a method for moving to a container location provided in an embodiment of the present application, refer to Figure 8 Shown, including:

[0160] S801: Based on the subtask of “moving to the container location”, the large language model determines that the service to be called is the entity control service, generates a third service response message, and sends it to the client.

[0161] S802: The client calls the main control service through the server to control the robot to move to the container position.

[0162] S803: The client returns the execution result of controlling the robot to move to the container position to the large language model.

[0163] S804: The large language model generates a third natural language response and returns it to the client.

[0164] S805: The client calls the TTS service through the server and returns the third voice feedback.

[0165] S806: The client plays the third voice feedback to the user.

[0166] Figure 9 A flow chart of a method for determining the position of a coke bottle provided in an embodiment of the present application is provided. Figure 9 Shown, including:

[0167] S901: Based on the subtask of “determining the location of the Coke bottle”, the large language model determines that the service to be called is the VLM service, generates a fourth service response message, and sends it to the client.

[0168] S902: The client calls the VLM service through the server to determine the location of the Coke bottle.

[0169] S903: The client sends the location of the Coke bottle to the large language model.

[0170] S904: The large language model generates a fourth natural language response and returns it to the client.

[0171] S905: The client calls the TTS service through the server and returns the fourth voice feedback.

[0172] S906: The client plays the fourth voice feedback to the user.

[0173] Figure 10A flow chart of a method for grabbing a cola bottle provided in an embodiment of the present application is provided. Figure 10 Shown, including:

[0174] S1001: Based on the subtask of "grabbing the Coke bottle", the large language model determines that the service to be called is the ontology control service, generates a fifth service response message, and sends it to the client.

[0175] S1002: The client calls the ontology control service through the server to grab the Coke bottle.

[0176] S1003: The client returns the result of grabbing the Coke bottle to the large language model.

[0177] S1004: The large language model generates a fifth natural language response and returns it to the client.

[0178] S1005: The client calls the TTS service through the server and returns the fifth voice feedback.

[0179] S1006: The client plays the fifth voice feedback to the user.

[0180] Figure 11 A flowchart of a method for determining the location of user A provided in an embodiment of the present application is provided. Figure 11 Shown, including:

[0181] S1101: Based on the subtask of “determining the location of user A”, the large language model determines that the service to be called is the face recognition service, generates sixth service response information, and sends it to the client.

[0182] S1102: The client calls the face recognition service through the server to determine the location of user A.

[0183] S1103: The client returns the location of user A to the large language model.

[0184] S1104: The large language model generates a sixth natural language response and returns it to the client.

[0185] S1105: The client calls the TTS service through the server and returns the sixth voice feedback.

[0186] S1106: The client plays the sixth voice feedback to the user.

[0187] Figure 12 A flow chart of a method for determining a driving path to user A provided in an embodiment of the present application, referring to Figure 12 Shown, including:

[0188] S1201: Based on the subtask of “determining the driving route to user A”, the large language model determines that the service to be called is the route planning service, generates a seventh service response message, and sends it to the client.

[0189] S1202: The client calls the path planning service through the server to determine the path from the robot's current position to user A.

[0190] S1203: The client returns the path from the robot's current location to user A to the large language model.

[0191] S1204: The large language model generates a seventh natural language response and returns it to the client.

[0192] S1205: The client calls the TTS service through the server and returns the seventh voice feedback.

[0193] S1206: The client plays the seventh voice feedback to the user.

[0194] Figure 13 A flow chart of a method for moving to the location of user A provided in an embodiment of the present application, refer to Figure 13 Shown, including:

[0195] S1301: Based on the subtask of “moving to the location of user A”, the large language model determines that the service to be called is the ontology control service, generates eighth service response information, and sends it to the client.

[0196] S1302: The client calls the main control service through the server to control the robot to move to the position of user A.

[0197] S1303: The client returns the execution result of controlling the robot to move to the position of user A to the large language model.

[0198] S1304: The large language model generates an eighth natural language response.

[0199] S1305: The client calls the TTS service through the server and returns the eighth voice feedback.

[0200] S1306: The client plays the eighth voice feedback to the user.

[0201] Execute the above Figures 6 to 13 The method shown can complete the user instruction of "go to the container and give the bottle of Coke to user A".

[0202] Figure 14 A schematic diagram of the structure of an embodied intelligent execution device provided in an embodiment of the present application Figure 1, applied to the client, the client is deployed on the edge side of the robot, the client registers multiple services, the services are called functions, and the multiple services communicate based on the model context MCP protocol; Figure 14 As shown, the embodied intelligent execution device 1400 provided in this embodiment includes:

[0203] Receiving module 1401, used to receive instructions input by the user;

[0204] a sending module 1402 configured to send the instruction and available service information to the large language model, so that the large language model generates service response information, wherein the available service information includes a plurality of services that can be called, and the service response information includes the service to be called;

[0205] The receiving module 1401 is further configured to receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, where the server is deployed on the edge side of the robot;

[0206] The sending module 1402 is further configured to send the service execution result to the large language model, so that the large language model generates a natural language response based on the service execution result;

[0207] The receiving module 1401 is further configured to receive the natural language response and display it to the user.

[0208] The embodied intelligent execution device provided in this embodiment can execute the above Figure 2 The implementation principle and technical effect of the method shown are similar, and will not be described in detail in this embodiment.

[0209] Figure 15 A schematic diagram of the structure of an embodied intelligent execution device provided in an embodiment of the present application Figure 2 , applied to a large language model, the large language model is deployed on a server; Figure 15 As shown, the embodied intelligent execution device 1500 provided in this embodiment includes:

[0210] Receiving module 1501, configured to receive user input instructions and available service information sent by a client, wherein the client has multiple services registered, each of which is a called function, and the multiple services communicate based on the Model Context MCP protocol, and the available service information includes multiple services that can be called;

[0211] The processing module 1502 is configured to generate service response information according to the instruction and the available service information, wherein the service response information includes the service to be called;

[0212] A sending module 1503 is configured to send the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, and the server is deployed on the edge side of the robot;

[0213] The receiving module 1501 is further configured to receive the service execution result fed back by the client;

[0214] The processing module 1502 is further configured to generate a natural language response according to the service execution result, and send the natural language response to the client.

[0215] The embodied intelligent execution device provided in this embodiment can execute the above Figure 4 The implementation principle and technical effect of the method shown are similar, and will not be described in detail in this embodiment.

[0216] Figure 16 This is a structural diagram of an embodied intelligent execution device provided in an embodiment of the present application. Figure 16 As shown, the embodied intelligent execution device 1600 provided in this embodiment includes: at least one processor 1601 and a memory 1602. Optionally, the device 160 further includes a communication component 1603. The processor 1601, the memory 1602, and the communication component 1603 are connected via a bus 1604.

[0217] During the specific implementation process, at least one processor 1601 executes the computer-executable instructions stored in the memory 1602, so that the at least one processor 1601 performs the above method.

[0218] The specific implementation process of the processor 1601 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0219] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules in the processor.

[0220] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0221] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0222] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0223] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0224] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0225] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0226] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0227] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0228] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0229] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0230] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0231] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. An embodied intelligence execution method, characterized in that: The method is applied to a client, the client is deployed on the edge side of the robot, the client registers multiple services, the services are called functions, and the multiple services communicate based on the model context MCP protocol; the method includes: Receive user input instructions; Sending the instruction and available service information to a large language model so that the large language model generates service response information, wherein the available service information includes a plurality of services that can be called, and the service response information includes the service to be called; Receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, where the server is deployed on the edge side of the robot; The service execution result is sent to the large language model so that the large language model generates a natural language response based on the service execution result, and the natural language response is received and displayed to the user.

2. The method according to claim 1, characterized in that The instruction includes a plurality of subtasks, wherein the plurality of subtasks are obtained by decomposing the instruction by the large language model, and each subtask has corresponding service response information; The receiving of service response information sent by the large language model includes: After displaying the natural language response corresponding to the current subtask to the user, service response information corresponding to the next subtask sent by the large language model is received.

3. The method according to claim 1 or 2, characterized in that The method further comprises: After the client is started, it sends a connection request to the server and receives confirmation of the connection from the server; Sending an available service information request to the server, and receiving available service information fed back by the server.

4. An embodied intelligence execution method, characterized in that: Applied to a large language model, the large language model is deployed on a server, and the method includes: Receiving a user input instruction and available service information sent by a client, wherein the client has multiple services registered, the services being called functions, and the multiple services communicating based on the Model Context MCP protocol, and the available service information including multiple services that can be called; Generate service response information according to the instruction and the available service information, wherein the service response information includes the service to be called; Sending the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, where the server is deployed on the edge side of the robot; Receive the service execution result fed back by the client, generate a natural language response according to the service execution result, and send the natural language response to the client.

5. The method according to claim 4, characterized in that Generating service response information according to the instruction and the available service information includes: The instruction is decomposed into tasks to obtain a plurality of subtasks, and a service corresponding to each subtask is determined according to the available service information to generate service response information corresponding to each subtask.

6. The method according to claim 5, characterized in that The sending the service response information to the client includes: After sending the natural language response corresponding to the current subtask to the client, service response information corresponding to the next subtask is sent to the client.

7. The method according to claim 5, characterized in that The available service information further includes: service description information of each service; and determining the service corresponding to each subtask according to the available service information includes: Determine the service corresponding to each subtask based on the prompt information and the service description information of each service; The prompt information includes a plurality of preset tasks and one or more services to be called for each task.

8. An embodied intelligent execution device, characterized in that: Applied to the client, the client is deployed on the edge side of the robot, the client registers multiple services, the services are called functions, and the multiple services communicate based on the model context MCP protocol; including: A receiving module, configured to receive instructions input by a user; a sending module, configured to send the instruction and available service information to the large language model, so that the large language model generates service response information, wherein the available service information includes a plurality of services that can be called, and the service response information includes the service to be called; The receiving module is further configured to receive service response information sent by the large language model, call the service to be called through the server, and receive the service execution result fed back by the server, wherein the server is deployed on the edge side of the robot; The sending module is further configured to send the service execution result to the large language model, so that the large language model generates a natural language response based on the service execution result; The receiving module is further configured to receive the natural language response and display it to the user.

9. An embodied intelligent execution device, characterized in that: Applied to a large language model, the large language model is deployed on a server; including: a receiving module, configured to receive user input instructions and available service information sent by a client, wherein the client has multiple services registered, each of which is a called function, and the multiple services communicate based on the Model Context MCP protocol, and the available service information includes multiple services that can be called; a processing module, configured to generate service response information according to the instruction and the available service information, wherein the service response information includes the service to be called; A sending module, configured to send the service response information to the client, so that the client calls the service to be called contained in the service response information through the server, wherein the server is deployed on the edge side of the robot; The receiving module is further configured to receive the service execution result fed back by the client; The processing module is further configured to generate a natural language response according to the service execution result, and send the natural language response to the client.

10. An embodied intelligent execution device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 3 or any one of claims 4 to 7.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 3 or any one of claims 4 to 7.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 3 or any one of claims 4 to 7 when executed by a processor.

Images