Intelligent interaction method and system for large language model and tool body in embedded system

Abstract

The invention provides a method and a system for interaction between a large language model and a body agent in an embedded system, and the method comprises the steps: an instruction sender sends an instruction to a central calculation unit and an information transmission unit of the body agent, generates a function name set and a formal parameter list which are needed by the execution of the body agent through the analysis and reasoning of the large language model, and transmits the function name set and the formal parameter list to the center calculation unit and the information transmission unit; the function name set and the formal parameter list are decoded and then sent to the execution mechanism of the body agent for execution, an embedded real-time operating system is used, the accuracy and the real-time performance of the execution mechanism of the body agent can be greatly improved, and meanwhile system debugging and maintenance are facilitated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and large language models, and in particular to a method and system for interaction between a large language model and embodied intelligence in an embedded system. Background Art

[0002] With the rapid development of science and technology, artificial intelligence (AI) has become one of the hottest topics in today's society. Among them, the combination of embodied intelligence and large language models has led a new round of technological innovation and brought unprecedented breakthroughs to the field of artificial intelligence. Embodied intelligence emphasizes the perception, understanding and action capabilities of intelligence. It relies on the interaction between physical entities and the environment to achieve more realistic and natural intelligent performance. The large language model refers to a deep learning model obtained through large-scale data training, which has powerful feature learning and pattern recognition capabilities. Combining embodied intelligence with a large language model can give full play to the advantages of both and open up new paths for the development of AI technology.

[0003] At present, there have been extensive studies on the integration of large language models with embodied intelligence, such as the prompt method based on Microsoft Research and the function calling method based on OpenAI, but all technical methods are based on the implementation of the Linux operating system. There is no efficient technical method for solutions for embedded real-time systems. However, in the future, hundreds of millions of embedded devices will gradually become intelligent. Considering technology, cost and other aspects, large language models and embodied intelligence deployment solutions based on embedded real-time operating systems will have huge advantages.

[0004] With the rapid development of deep learning technology, various types of large language models have emerged one after another. Whether they are open source models or closed source models, the expressiveness of large language models is getting stronger and stronger. However, large language models themselves still cannot break through the key problem of real-time interaction with the real environment. Embodied intelligence emphasizes the real-time interaction ability of intelligent systems and the environment, which is a very beneficial supplement to large language models in this field. This intelligent agent can interact with the environment through sensors and actuators, and make corresponding decisions and actions according to changes in the environment.

[0005] The existing technical methods for large language models and embodied intelligence interaction are all based on the Linux operating system. Most of these methods are deployed on chip resources with better resources. However, for some cost-sensitive embedded scenarios, this technical method lacks competitive advantages in terms of cost and low power consumption. In this case, embedded real-time operating systems have a higher cost-effectiveness. However, the current technical methods basically do not involve technical methods in this field. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method and system for interaction between a large language model and embodied intelligence in an embedded system.

[0007] The present invention is achieved through the following technical solutions.

[0008] The present invention provides a method for interacting a large language model with embodied intelligence in an embedded system, comprising the following steps: Step 1: The instruction sender sends instructions to the central computing unit and information transmission unit of the embodied intelligent body, and generates the function name set and parameter list required for the embodied intelligent body to execute through large language model analysis and reasoning; Step 2: The function name set and parameter list required for the embodied intelligent agent to execute are transmitted to the embedded end through the robot, and the embedded end decodes the function name set and parameter list; Step 3: Obtain a function with a corresponding function name in the embodied agent execution function library, wherein the function is an embodied agent action instruction, and execute the action instruction; Repeat steps 1 to 3 to receive, parse and execute the execution function flow of the embodied intelligent body in turn to complete the instructions issued by the instruction sender.

[0009] Furthermore, the function name set and parameter list required for the embodied agent to execute are transmitted to the embedded end through the robot, and the embedded end decodes the function name set and parameter list including: The robot obtains relevant instructions, and the data structure of the relevant instructions is a custom structure; The information decoding module parses the specific content of the relevant instructions, which includes a function name set and a parameter list; The function query module searches for the embodied agent execution function library compiled in the embedded real-time operating system RT-Thread OS. If the corresponding function is found, the relevant code of the function is executed. If not, the information that the relevant instruction is not found is returned to the server. The relevant code drives the specific functions of the embodied agent.

[0010] Furthermore, the functions for obtaining the corresponding function names in the embodied agent execution function library include: Put the functions in the same memory segment, the memory segment stores global variables of the same type, and the global variables store the function entry; The global variables in the memory segment are traversed one by one, and if a matching function is found, the matching function is executed.

[0011] Furthermore, global variables are function pointer variables.

[0012] At the same time, the present invention also provides a large language model and embodied intelligent interaction system in an embedded system, which is characterized by comprising a large language model, a robot, and an embedded real-time operating system, wherein the large language model and the robot communicate with the embedded real-time operating system respectively through the Internet protocol.

[0013] Furthermore, the large language model and the robot are deployed on the server side, and the embedded real-time operating system is deployed on the embedded side. The beneficial effect of the present invention is that by designing an information decoding module and a function search module, it is possible to effectively search for execution functions available to the embodied intelligent body, and ultimately execute related actions on the embedded end.

[0014] Using the ROS2 distributed system, the central computing and information transmission unit of the embodied intelligent body is separated from the embodied intelligent body actuator, which is conducive to achieving the purpose of one central computing and information transmission unit controlling multiple embodied intelligent body actuators, effectively improving the reuse rate of the central computing and information transmission unit, and reducing the overall embodied intelligent body cost.

[0015] The use of embedded real-time operating systems can greatly improve the accuracy and real-time performance of embodied intelligent body actuators, and is also beneficial for system debugging and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow chart of the method of the present invention; Figure 2 It is a schematic diagram of decoding the function name set and the parameter list of an embodiment of the present invention; Figure 3 This is a schematic diagram of returning to the server side after the instruction of the embodiment of the present invention; Figure 4 It is a schematic diagram of an execution function library initialization method of an embodied intelligent body according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below, but the scope of protection claimed is not limited to the description.

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] This embodiment takes the embodied intelligent vehicle as an example to describe the structure of the large language model and embodied intelligent agent interaction design method in the entire real-time operating system.

[0020] In this embodiment, the Windows operating system is used on the server side to obtain more support packages for the Python scripting language. The embedded real-time operating system running on the embedded side is the RT-Thread operating system, and the open source large language model chatglm3 is installed on the server side. The model provides a native tool call that can provide a function call function. According to the json file format provided by the tool call, the basic functions of the embodied intelligent chassis operation are designed, including four mechanism operation functions such as forward, backward, left turn and right turn, each of which includes the embodied intelligent chassis operation function name, function description and parameter list required by the function, where: The function names and functional descriptions corresponding to the forward operation include: { "name": "go_ahead", "description": "The car moves forward", "parameters": { "type": "object", "properties": { "distance": { "description": "Distance traveled, in meters" } }, "required": [ "distance" ] } }; The function names and functional descriptions corresponding to the back operation include: { "name": "back", "description": "The car moves backward", "parameters": { "type": "object", "properties": { "distance": { "description": "The distance of retreat, in meters" } }, "required": [ "distance" ] } }, The function names and functional descriptions corresponding to the left turn operation include: { "name": "turn_left", "description": "The car turns left", "parameters": { "type": "object", "properties": { "angle": { "description": "Left turn angle, in degrees" } }, "required": [ "angle" ] } }, The function names and functional descriptions corresponding to the right turn operation include: { "name": "turn_right", "description": "The car turns right", "parameters": { "type": "object", "properties": { "angle": { "description": "Right turn angle, in degrees" } }, "required": [ "angle" ] } }, During use, the server inputs the command "Please drive forward 5 meters, then turn left 90 degrees, and then drive 9 meters" to the large language model input port. After analysis by the chatglm3 large language model, three groups of functions are finally returned at once: 1.{"name": "go_ahead", "parameters": {"requried": "5"}}; 2.{"name": "turn_left", "parameters": {"requried": "90"}}; 3.{"name": "go_ahead", "parameters": {"requried": "9"}}; The three groups of functions are passed to the embodied intelligent body's actuator in turn through the ROS2 robot distributed communication mechanism. In the actuator, the ROS2 robot terminal in the embedded real-time system receives and parses the relevant instructions. The function query module is used to find the implementation of the three groups of functions on the embedded end in the execution function library, and the execution function library is initialized. The initialization process is as follows: Figure 4 As described above, a section of address is opened in the memory code segment, and the first address is __embedAItab_start, which is the starting position of the execution function library of the embodied intelligent body. And __embedAItab_end is the end position of the execution function library of the embodied intelligent body. During the compilation process, all functions are compiled into the operating system. The execution function library includes execution functions such as void go_ahead (float distance), void back (float distance), void turn_left (float angle), and void turn_right (float angle). After executing the corresponding instructions, "the command has been completed" is returned to the server side, thereby completing the overall command execution work.

[0021] The above embodiments are preferred solutions for implementing the present invention. It should be noted that any obvious replacements and minor changes are within the protection scope of the present invention without departing from the concept of the present invention.

Claims

1. A method for interaction between a large language model and embodied intelligence in an embedded system, characterized in that include: Step 1: The instruction sender sends instructions to the central computing unit and information transmission unit of the embodied intelligent body, and generates the function name set and parameter list required for the embodied intelligent body to execute through large language model analysis and reasoning; Step 2: The function name set and parameter list required for the embodied intelligent agent to execute are transmitted to the embedded end through the robot, and the embedded end decodes the function name set and parameter list; Step 3: Obtain a function with a corresponding function name in the embodied agent execution function library, wherein the function is an embodied agent action instruction, and execute the action instruction; Repeat steps 1 to 3 to receive, parse and execute the execution function flow of the embodied intelligent body in turn to complete the instructions issued by the instruction sender.

2. The method for interaction between a large language model and embodied intelligence in an embedded system as claimed in claim 1, characterized in that: The function name set and parameter list required for the embodied intelligent agent to execute are transmitted to the embedded end through the robot, and the embedded end decodes the function name set and parameter list, including: The robot obtains relevant instructions, and the data structure of the relevant instructions is a custom structure; The information decoding module parses the specific content of the relevant instructions, which includes a function name set and a parameter list; The function query module searches for the embodied agent execution function library compiled in the embedded real-time operating system RT-Thread OS. If the corresponding function is found, the relevant code of the function is executed. If not, the information that the relevant instruction is not found is returned to the server. The relevant code drives the specific functions of the embodied agent.

3. The method for interaction between a large language model and embodied intelligence in an embedded system as claimed in claim 1, characterized in that: The function of obtaining the corresponding function name in the embodied agent execution function library includes: Put the functions in the same memory segment, the memory segment stores global variables of the same type, and the global variables store the function entry; The global variables in the memory segment are traversed one by one, and if a matching function is found, the matching function is executed.

4. A method for interaction between a large language model and embodied intelligence in an embedded system as claimed in claim 1 or 3, characterized in that: The global variable is a function pointer variable.

5. A large language model and embodied intelligence interaction system in an embedded system, characterized in that: It includes a large language model, a robot, and an embedded real-time operating system. The large language model and the robot communicate with the embedded real-time operating system respectively through the Internet protocol.

6. A large language model and embodied intelligence interaction system in an embedded system as claimed in claim 5, characterized in that: The large language model and the robot are deployed on the server side, and the embedded real-time operating system is deployed on the embedded side.

Images