Intelligent agent robot with semi-autonomous exploration capability
By integrating sensors and guided engine modules in intelligent unmanned devices, the integration of autonomous task execution and user instructions is solved, and the problem of insufficient autonomous exploration capabilities under communication restrictions in extreme environments is improved, and the efficiency of robots in deep sea and outer space exploration is improved. and security.
Patent Information
- Application Number
- CN202410946234.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-15
AI Technical Summary
In extreme environments such as deep sea and outer space, remote-controlled underwater robots (ROVs) and autonomous underwater unmanned aerial vehicles (AUVs) are difficult to achieve efficient autonomous exploration and mission execution due to communication delay and quality limitations, resulting in semi-autonomous exploration. The need for capability is urgent.
Design an intelligent unmanned device that integrates sensors, hardware modules, guide engine modules and intelligent proxy modules, which can independently complete conservative and high-complete tasks before the user's high-priority commands have arrived, and receive user instructions. Post-adjustment and optimization of task execution. Data is collected through sensors, the guide language engine module generates autonomous guide language and user guide language, and the intelligent agent module processes commands asynchronously to realize task disassembly and execution.
It improves the work efficiency and safety of robots in extreme environments, reduces their dependence on real-time communication, provides new possibilities for deep-sea and outer space exploration, and enhances the level of intelligence of robots in unknown environments.
Smart Images

Figure CN118990526A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of natural language processing and intelligent robot technology, and in particular to a robot with intelligent agent capabilities. Background Art
[0002] The Large Language Model (LLM) is good at interacting with users in the form of dialogue due to its powerful natural language processing capabilities. When a user enters a prompt as input, LLM can not only capture key information from personalized context information through its built-in attention mechanism, but also extract relevant information from a broad and compressed knowledge base, thereby generating high-quality, targeted answers and returning them to the user.
[0003] When this interaction is no longer limited to a single natural language modality, but extends to multiple modalities including pictures, videos, sounds, etc., the LLM that can process and generate these multimodal information is called a multimodal large language model. This model can understand and generate information across different media, greatly enriching the dimension and depth of human-computer interaction.
[0004] A module with multimodal large model capabilities that can actively interact with people, machines or the environment is called a multimodal agent. A multimodal agent not only understands and responds to inputs from different modalities, but also generates corresponding multimodal outputs to adapt to different interaction scenarios, thereby providing a richer, more intuitive and immersive interactive experience.
[0005] Intelligent Agent is a broader concept that covers multimodal agents and other intelligent systems that can act autonomously, perceive the environment, make decisions and interact with it. The core of intelligent agents lies in their intelligence, that is, they can make reasonable and effective responses in uncertain and dynamic environments based on complex algorithms and models.
[0006] The integration of multimodal large models and multimodal agent technology has significantly enhanced the ability of robots to perform complex tasks. Take the Mobile ALOHA project jointly launched by Google DeepMind and Stanford University in 2024 as an example. The intelligent robots in this project can easily cope with various tasks such as cooking (including delicate operations such as frying steaks and beating eggs) and daily housework (folding quilts and sweeping the floor) with their advanced technology, showing a high degree of intelligence and flexibility. In the field of autonomous driving, Tesla's Full Self-Driving (FSD) system has achieved deep integration and analysis of diversified data from vehicle sensors by introducing multimodal large models, so as to make accurate decisions in complex road environments and plan the optimal driving route, greatly improving the safety and reliability of autonomous driving. The field of unmanned aerial vehicles (UAVs) has also benefited from the development of large model technology. Microsoft's ChatGPT for Robotics project is a typical example. The project uses advanced large model technology to enable drones to handle complex tasks, such as autonomously exploring and identifying goods along specific paths (such as Z-shaped folding paths), demonstrating the huge potential of drones in logistics, warehousing and other fields.
[0007] In the field of remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs), although these devices play an important role in ocean exploration, underwater construction, underwater archaeology, etc., there is still a gap in the integration with large language models. Faced with extreme environments such as deep-sea exploration and distant planets, the limitations of communication latency and communication quality have become a major challenge. For example, there is a liquid ocean under the ice shell of Europa, which is very worth exploring remotely for life. However, the average distance between Europa and the earth is about 600 million kilometers, and the delay of a radio communication between them is about half an hour. In this context, the demand for semi-autonomous exploration capabilities of drones and underwater robots is particularly urgent.
[0008] The patent of this invention is an innovative solution to this demand. By giving ROV and AUV a certain degree of autonomous exploration capability, they can autonomously complete a series of conservative and high-completion tasks before the user's instructions arrive. Once the user's instructions arrive, the system can make necessary adjustments and optimizations to the task execution according to the new instructions. This design not only improves the working efficiency and safety of robots in extreme environments, but also reduces the dependence on real-time communication, providing new possibilities for the exploration of unknown environments such as the deep sea and outer space.
[0009] The following are the comparative patent search results related to this patent technology. These patents involve robot autonomous exploration, multimodal data processing or similar technical fields to varying degrees, providing reference and comparison for the innovation and practicality of this invention.
[0010] "Basic Multimodal Agent Interaction", CN 118103116 A, 2024.05.28, Microsoft Technology Licensing LLC. This patent focuses on human-computer interaction and designs a cyclic interaction process between users, multimodal agents, and machines. The application scenario is mainly computer games. Our innovation lies in: the robot autonomously triggers an event during the exploration process. This event generates a user guide and interacts with the user at the same time, while generating an autonomous guide. The autonomous guide enables the robot to autonomously analyze and process the event, giving the robot more self-decision-making power and intelligence. Finally, we designed a fusion strategy when autonomous instructions and user instructions exist at the same time, which solves the potential contradiction between machine self-intelligence and user decision-making.
[0011] "Natural language control of a robot", WO2023192497A1 & US20230311335A1, 2023-10-05, Google LLC. The patent designs the following cyclic process: the user's verbal expression instructions and the completed subtask information are enhanced by the guide language engine to obtain a quote that is more inspiring for the big model to answer, and the quote is fed into the big model for downstream task decomposition and execution. The guide language and subsequent processes designed in this patent are only for machines. The difference of this patent is that the robot autonomously triggers the guide language engine during the exploration process to obtain a machine-oriented guide language and a user-oriented guide language. For this, this patent also newly designs a corresponding asynchronous fusion processing strategy downstream.
[0012] "Mind map prompting enhanced basic model task decomposition and scheduling method and device", CN117370638B, 2024-05-07, Institute of Space Information Innovation, Chinese Academy of Sciences. This patent proposes a method for decomposing complex tasks into simple subtasks. This method is achieved by local search of multiple thought chains, which is only a feasible way to decompose complex tasks into simple subtasks. The widely used chain-of-thought prompting (CoT promtping) technology and the open source Langchain plug-in developed based on the chain-of-thought prompting technology can also achieve the decomposition of complex tasks. Summary of the invention
[0013] The present invention proposes an innovative method for user-robot interaction, which aims to significantly improve the intelligence level of the robot in performing tasks in unknown environments. The robot referred to in this method is an intelligent unmanned device with semi-autonomous exploration capabilities, which integrates core components such as sensors, hardware modules, storage modules, guide language engine modules and intelligent agent modules. The robot's guide language engine module, intelligent agent module and storage module can be deployed on the robot body, and can also be flexibly configured on near-end, remote or cloud computing devices to achieve communication with the robot through efficient wired or wireless communication links.
[0014] The semi-autonomous exploration capability of the robot is reflected in that it allows the robot to autonomously issue and complete some conservative and high-completion tasks before the user's high-priority command arrives. Once the user's command arrives, the robot can quickly adjust and optimize the task being executed accordingly, thereby ensuring the accuracy and efficiency of task execution.
[0015] On the one hand, the present invention provides a method for generating autonomous guidance words and user guidance words based on external environment triggers, including: collecting environmental and self-state data through robot sensors, and after preliminary internal processing, handing them over to the guidance language engine module for further analysis; the guidance language engine module autonomously determines whether to trigger a new event based on the received information; once it is determined that a new event has been triggered, the guidance language engine will comprehensively consider the current environmental information, contextual data and guidance language database to generate two guidance words: one is an autonomous guidance language, which is directly input into the intelligent agent module to generate preliminary autonomous task commands; the other is a user guidance language, which is provided to the user to feedback the current environmental conditions and guide the user to participate in the decision-making process.
[0016] According to an embodiment of the present invention, during the exploration process of the underwater robot, it receives data from sensors in real time, including but not limited to images from cameras, terrain and structure from sonar and lidar, acceleration from inertial measurement units, water depth from pressure sensors, direction of the geomagnetic field from magnetometers, etc.
[0017] According to an embodiment of the present invention, these data are first processed by the robot's internal processor, such as data sampling, outlier filtering, and data format conversion, and then the data in a specific format is passed to the guide language engine for in-depth analysis.
[0018] According to an embodiment of the present invention, the guide language engine module will autonomously determine whether to trigger a new event based on the received multimodal data. For example, the pre-processed image will enter the multimodal language model in the guide language engine, and the model will perform multi-target detection on the image and output the possible classification and corresponding probability of the identified target. When the model detects the corresponding target category and the predicted probability reaches the threshold, a new event is triggered.
[0019] According to an embodiment of the present invention, once a new event is determined to be triggered, the guide language engine module will integrate the current environment information, context data and guide language database. The current environment information refers to the detected target information, self-state data, etc.; the context data includes long-term memory data, such as the professional knowledge of the exploration field, and also includes short-term memory data, such as the historical conversation records between the user and the robot; the guide language database refers to a set of pre-written guide language paradigms, which can be integrated into the final guide language, so as to better help the language model make predictions.
[0020] According to an embodiment of the present invention, the guide language engine module generates two guide languages: autonomous guide language and user guide language. Autonomous guide language is oriented to the robot itself, and it is directly input to the intelligent agent module to generate preliminary autonomous task commands; user guide language is oriented to the user, and it needs to feedback the current environmental conditions, the recognition results of the model, and the problems that the machine itself cannot determine, and guide the user to participate in the top-level decision-making of the task. In summary, the different communication objects of autonomous guide language and user guide language determine the obvious differences in their content, format and tone in natural language expression.
[0021] On the other hand, the present invention provides an execution method for asynchronously fusing several natural language commands of different priorities, including: an intelligent agent module asynchronously receives autonomous guidance words from a guidance word engine and high-priority commands from users; the intelligent agent module generates autonomous commands based on the autonomous guidance words; after receiving autonomous commands or high-priority commands from users, the intelligent agent will conduct a comprehensive evaluation based on memory, and fuse and generate a sub-task sequence described in natural language based on the priority of the commands; these sub-task sequences will then be converted by the intelligent agent module into codes, API sequences or call sequences of different sub-models that can be executed by robot hardware; finally, these instructions are sent to the corresponding hardware platform or module and executed in sequence.
[0022] According to an embodiment of the present invention, after receiving the environment information and the user guide, the user can make a decision based on the information and output a high priority command in a spoken form. In addition, the user has the right to freely issue a high priority command at any time to adjust the task execution.
[0023] According to an embodiment of the present invention, the intelligent agent module asynchronously processes the autonomous guidance words from the guidance words engine and the high-priority commands of the user. The autonomous guidance words and the high-priority commands can be stored in different message queues in the memory respectively.
[0024] According to an embodiment of the present invention, after receiving the autonomous guidance language, the language model in the intelligent agent module will combine the long-term and short-term memory data in the memory to perform reasoning and output the corresponding autonomous command. The autonomous command is further broken down by the language model into multiple subtasks described in natural language. These subtasks are further converted by the language model into codes, API sequences or call sequences of different submodels that can be executed by the robot hardware. Finally, these instructions are sent to the corresponding hardware platform or module and executed in sequence.
[0025] According to an embodiment of the present invention, the code, API sequence or calling sequence of different sub-models being executed by the robot hardware will be used as the current task execution status information and transmitted back to the memory in real time.
[0026] According to an embodiment of the present invention, after receiving a high-priority command from a user, the intelligent agent module will also combine the long-term and short-term memory data in the memory, call the language model, and generate a subtask sequence that combines the natural language description of the high- and low-priority commands. Note that since the status information of the task execution has been transmitted back to the memory, the subtask sequence and execution code output by the language model will update the part that has not yet been executed, and will not change the part that has already been executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] For a more complete understanding of the present invention and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:
[0028] Figure 1 The present invention shows a general flow chart of an intelligent agent robot executing autonomously and interacting with a user, provided by an embodiment of the present invention.
[0029] Figure 2A The autonomous guidance language processing part of the intelligent agent system interaction example provided by the embodiment of the present invention is shown.
[0030] Figure 2B The user guide language processing part of the intelligent agent system interaction example provided by the embodiment of the present invention is shown.
[0031] Figure 3 A signaling interaction diagram of an intelligent agent system provided by an embodiment of the present invention is shown.
[0032] Figure 4 The figure shows the structure diagram of the intelligent agent robot system provided by the embodiment of the present invention.
[0033] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0034] It should be clear that the specific embodiments listed here are only used to illustrate the concept of the present invention and are not intended to limit the present invention in any form. In the subsequent detailed description, in order to more fully understand the present invention, we provide many specific details. However, it is worth noting that even without these detailed details, one or more embodiments of the present invention can still be effectively implemented.
[0035] This specification contains flow charts and block diagrams to assist understanding. It should be understood that some of the blocks or combinations thereof in these diagrams can be implemented in practice by computer programs. These programs can be designed to run on a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby creating a device capable of performing the functions / operations described in the diagrams.
[0036] Furthermore, the technical implementation of the present invention can be in the form of hardware or software (including firmware, microcode, etc.), or a combination of the two. In addition, the technology of the present invention can also exist in the form of a computer program product, which is stored on a computer-readable medium for use alone or in combination with an instruction execution system. In this context, a computer-readable medium is broadly defined as any medium that can carry, store, transmit, disseminate or transfer instructions. For example, this includes but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices, components or communication media. Specifically, examples of computer-readable media include magnetic storage devices (such as magnetic tapes, hard disk drives HDD), optical storage devices (such as optical disks CD-ROM), memory devices (such as random access memory RAM, flash memory), and wired or wireless communication links.
[0037] Figure 1 The present invention shows a general flow chart of an intelligent agent robot executing autonomously and interacting with a user, provided by an embodiment of the present invention.
[0038] like Figure 1 As shown, an embodiment of the present invention provides a method for integrating autonomous execution of an intelligent agent robot and interactive execution with a user, including S110 to S150.
[0039] S110, the robot sensor receives the surrounding environment information and transmits the processed information to the guidance language engine.
[0040] S120, the guide engine autonomously determines whether to trigger a new event based on the input information. If triggered, the guide engine will combine the context data, guide data, and environmental information to output an autonomous guide and a user guide respectively.
[0041] S130, the autonomous guidance language is directly input into the intelligent agent to obtain an autonomous task command; after the user receives the environmental information and the user guidance language, he can decide to output a verbally expressed high-priority command, and the user can also freely issue a high-priority command at any time.
[0042] S140, the intelligent agent asynchronously receives autonomous task commands and high priority commands. Whenever a command is received, the intelligent agent outputs an updated final task command. Then, the intelligent agent disassembles the final task command and outputs it as a collection of multiple subtasks.
[0043] S150, the intelligent agent further converts the subtask set described in natural language into codes that can be executed by robot hardware, or API sequences, or call sequences of different sub-models. Finally, it is sent to the corresponding platform for execution in sequence.
[0044] The semi-autonomous exploration scenario of a robot in an unknown environment requires the robot to have a certain degree of autonomous reasoning ability based on changes in the external environment. It also requires the robot to be able to dynamically integrate the user's command information to instantly correct its own execution plan. Figure 1 The fusion method is a solution for this scenario. By giving robots a certain degree of autonomous exploration capabilities, they can autonomously complete a series of conservative and high-completion tasks before the user's instructions arrive. Once the user's instructions arrive, the system can make necessary adjustments and optimizations to the task execution according to the new instructions, thereby ensuring the user's top-level control over the exploration task.
[0045] The following will combine Figure 2 ~ Figure 4 A method for integrating autonomous execution and interactive execution with users by an intelligent agent robot provided in an embodiment of the present invention is described in detail.
[0046] Figure 1 The overall flow chart of FIG. 1 can be divided into two parts: an autonomous guidance language processing part and a high priority command processing part according to the asynchronous reception of the autonomous task command and the high priority command in S140. Figure 2A The autonomous guidance language processing part of the intelligent agent system interaction example provided by the embodiment of the present invention is shown. Figure 2B The user guide language processing part of the intelligent agent system interaction example provided by the embodiment of the present invention is shown.
[0047] like Figure 2A As shown, when the robot performs a task, it receives data from sensor 201 in real time, including but not limited to images from the camera, terrain and structure from sonar and lidar, acceleration from the inertial measurement unit, atmospheric pressure from the barometric sensor, water depth from the pressure sensor, and direction of the geomagnetic field from the magnetometer.
[0048] These data are first processed by the robot's internal processor, such as data sampling, outlier filtering, and data format conversion, and then the data in a specific format is passed to the guide language engine 202 for in-depth analysis.
[0049] The guide language engine module 202 will autonomously determine whether to trigger a new event based on the received multimodal data. For example, the pre-processed camera image will enter the multimodal language model in the guide language engine 202, and the model will perform multi-target detection on the image and output the classification and corresponding probability of the identified target. When the model detects that the predicted probability of the coral reef and garbage categories reaches the threshold, a new event is triggered.
[0050] Once a new event is determined to be triggered, the guide engine module 202 will call the environmental information, context data and guide database in the memory 203. Environmental information refers to the historical data recorded by the sensor 201, its own state data, etc.; context data includes long-term memory data, such as professional knowledge in the field of underwater exploration, and short-term memory data, such as historical conversation records between users and robots; the guide database refers to a set of pre-written guide paradigms, such as: "Please make further judgments based on...", "...Tell me how to take action next?", "...A self-directed task instruction needs to be formulated", etc. These guide paradigms can be integrated into the final guide, so as to better help the language model make predictions.
[0051] The guide language engine module 202 generates two guide language: autonomous guide language 206 and user guide language 205. Note that the guide language engine module 202 can use a variety of technologies and models such as image processing, natural language processing, machine learning, dialogue management, knowledge base and rule engine to generate autonomous guide language 206 and user guide language 205. The guide language engine module of this embodiment adopts a multimodal language large model.
[0052] exist Figure 2AIn the example, autonomous guidance 206 is: "An elliptical object is identified on the seabed 5 meters ahead, which may be a coral reef or garbage. An autonomous task instruction needs to be formulated." User guidance 205 is: "I have identified an elliptical object on the seabed 5 meters ahead, and preliminarily judged it to be a coral reef or garbage. Please make a further judgment based on environmental information such as image sonar, and tell me what action to take next?" Autonomous guidance 206 is oriented to the robot itself, and it is directly input to the intelligent agent module 204 to generate a preliminary autonomous task command 207. User guidance 205 is oriented to the user, and it needs to feedback the current environmental conditions, the recognition results of the model, and the problems that the machine itself cannot determine, and guide the user to participate in the top-level decision-making of the task. In summary, the autonomous guidance 206 and the user guidance 205 have different communication objects, which determines the obvious differences in their content, format and tone in natural language expression.
[0053] Autonomous guide words 206 and user guide words 205, autonomous commands 207, subtask set 208, execution code 209, and Figure 2B The user high priority commands 210 in the message queue can be stored in different message queues in the memory respectively, and read in real time by the intelligent agent 204 and the guide language engine 202. The transmission and storage of this real-time stream data can be realized by the Kafaka message queue system.
[0054] The intelligent agent module 204 asynchronously processes the autonomous guidance words 205 from the guidance words engine 202 and the user's high priority command 210 .
[0055] exist Figure 2A In the example, after receiving the autonomous guidance 206, the language model in the intelligent agent module 204 will combine the long-term and short-term memory data in the memory 203 to perform reasoning and output the corresponding autonomous command 207, corresponding to the arrow 1 drawn by the intelligent agent 204. The autonomous command 207 is further decomposed by the language model into a subtask sequence 208A described in natural language, corresponding to the arrow 2 drawn by the intelligent agent 204. These subtasks are further converted by the language model into codes, API sequences or call sequences of different submodels that can be executed by the robot hardware, recorded as execution scripts 209A, corresponding to the arrow 3 drawn by the intelligent agent 204. Finally, these instructions are sent to the robot hardware platform 210 for execution in sequence, which corresponds to the arrow 4 drawn by 209A.
[0056] exist Figure 2AIn the example, the autonomous command 207 is: "Get closer to observe and further determine the properties of the object". This is a conservative task command with a high probability of successful execution. It is reasonable for the robot to execute this autonomous command before receiving the user's high-level command. The subtask sequence 208A is: "Sail 4 meters straight ahead; circle and take photos of the object; estimate the size and properties of the object", which describes three subtasks that need to be executed in sequence. The execution script 209A is: "# Sail 4 meters straight ahead\n inertia_goto(d_north=4, d_east=0)\n # Circle and take photos of the object\n; images = circle_photo()\n # Estimate the size and properties of the object\n res = estimate(images)\n". inertia_goto, circle_photo and estimate in the code are predefined application programming interfaces (APIs). The most fine-grained part of the program code that needs to be executed by the robot hardware can be sent to the robot hardware 210 for execution by the intelligent agent module through a specific message transmission protocol, such as MAVLink (Micro Air VehicleLink).
[0057] exist Figure 2A In the example, the script 209A is being executed line by line by the intelligent agent 204. At this time, the "inertia_goto" statement is being executed. The intelligent agent 204 will write the code statement and the corresponding subtask into the corresponding message queue in the memory. The intelligent agent 204 will also receive the execution result of "inertia_goto" from the robot hardware 210 through the message transmission protocol, and write the execution result into the corresponding message queue in the memory.
[0058] like Figure 2B As shown, after receiving the user guidance 205, the user retrieves the image data in the memory, identifies that the object is likely to be plastic waste, and combines other environmental data to make an independent decision and issue a high priority command 210. The content of the high priority command 210 is: "This object is a piece of plastic waste and can be picked up."
[0059] exist Figure 2BIn the example, the high priority command 210 is input into the intelligent agent module 204, corresponding to the arrow 1 introduced by the intelligent agent 204. The language model in the intelligent agent module 204 will combine the long and short-term memory data in the memory 203 to perform reasoning and generate a subtask sequence 208B that integrates the high and low priority commands, corresponding to the arrow 2 introduced by the intelligent agent 204. The content of the subtask sequence 208B described in natural language is: "Sail 4 meters straight ahead; circle and take pictures of the object; pick up the object and put it in the garbage bag". Note that Figure 2A Compared with the subtask sequence 208A, 208B replaces the original third subtask "estimate the size and attributes of the object" with "pick up the object and put it in the garbage bag", while the first two subtasks remain the same. This is because, Figure 2A The "inertia_goto" code of 209A has been executed, and the execution record has been stored in the memory. When the intelligent agent 204 sends 208A, it has learned the execution record of the "inertia_goto" code. In addition, the language model of the intelligent agent 204 integrates the information of the autonomous command 207 and the high-priority command 210, and infers that the first two subtasks "sail 4 meters straight ahead; circle and take pictures of the object" still need to be retained and executed, and only the third subtask needs to be replaced from "estimate the size and attributes of the object" to "pick up the object and put it in the garbage bag".
[0060] exist Figure 2B In the example, the subtask sequence 208B is further converted by the language model into a code, API sequence or call sequence of different sub-models that can be executed by the robot hardware, which is recorded as the execution script 209B, corresponding to the arrow 3 drawn by the intelligent agent 204. The content of the execution script 209B is: "# Sail 4 meters straight ahead\n inertia_goto(d_north=4, d_east=0)\n# Circle and take pictures of the object\n; images = circle_photo()\n # Estimate the size and attributes of the object\nres = pick() \n". Note that, unlike Figure 2A Compared with the execution script 209A, 209B replaces the original code corresponding to the third subtask "res = estimate(images)" with "res = pick()". Finally, the execution script 209B will be sent to the robot hardware platform 210 for execution in sequence, which corresponds to arrow 4 drawn from 209B.
[0061] exist Figure 2BIn the example, script 209B is being executed line by line by intelligent agent 204. At this time, the "circle_photo" statement is being executed. Intelligent agent 204 will write the code statement and the corresponding subtask into the corresponding message queue in the memory. Intelligent agent 204 will also receive the execution result of "circle_photo" by robot hardware 210 through the message transmission protocol, and write the execution result into the corresponding message queue in the memory.
[0062] Figure 3 A signaling interaction diagram of an intelligent agent system provided by an embodiment of the present invention is shown.
[0063] exist Figure 3 In the embodiment, when the robot is performing a task, its sensor 301 receives various types of sensor data in real time, including but not limited to images from cameras, terrain and structure from sonar and lidar, acceleration from inertial measurement unit, atmospheric pressure from air pressure sensor, water depth from pressure sensor, direction of geomagnetic field from magnetometer, etc.
[0064] These sensor data are initially processed by the robot's internal processor, and the robot's internal processor outputs a decision on whether to trigger a new event. Note that the robot's internal processor can be deployed in the sensor 301, or in the guidance language engine 302, or in other internal modules of the robot.
[0065] After receiving the message of the triggering event, the guide engine 302 sends the autonomous guide to the intelligent agent 303 and also sends the advanced guide to the user 305. Note that there is no requirement for the order in which the autonomous guide and the advanced guide are sent. Figure 3 Only one order is shown.
[0066] After receiving the autonomous guidance, the intelligent agent 303 generates an autonomous task command, and then further decomposes the autonomous task command into a subtask sequence and a corresponding execution script, and sends them line by line to the robot hardware platform 304 for execution.
[0067] After receiving the high-level guidance from the guidance engine 302, the user can freely choose whether to send and send several high-level task instructions to the intelligent agent 303 in combination with the environmental information and robot status information.
[0068] The intelligent agent 303 will receive the user's high-level task instructions, update the generated sub-task sequence and the corresponding execution script, and then send the new execution script line by line to the robot hardware platform 304 for execution.
[0069] Notice Figure 3What is shown is only one way for the intelligent agent 303 to asynchronously receive autonomous guidance words and high-level task instructions. In actual task scenarios, there are also situations where: the intelligent agent 303 continuously receives autonomous guidance words; the intelligent agent 303 continuously receives high-level task instructions; and the intelligent agent 303 receives autonomous guidance words of new events after receiving high-level task instructions. These correspond to claims 3, 4, 5, and 6.
[0070] Figure 4 The figure shows the structure diagram of the intelligent agent robot system provided by the embodiment of the present invention.
[0071] exist Figure 4 In the embodiment, the robot body 400 includes a robot control system 402, one or more operating components 403a-403m and one or more sensors 401a-401n.
[0072] Sensors 401a-401n may include visual sensors, light sensors, pressure sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although sensors 401a-401n are depicted as being integral to robot 400, this is not a limitation. In some implementations, sensors 401a-401n may be located external to robot 400, such as as a stand-alone unit.
[0073] The operating components 403a-403m may include one or more end effectors and / or one or more servo motors or other actuators for achieving movement of one or more components of the robot. For example, the robot 400 may have multiple degrees of freedom, and each actuator may control the movement of the robot 400 in one or more degrees of freedom according to a control command. Herein, the term "actuator" includes a mechanical or electrical device (such as a motor) that creates movement, as well as any driver associated with the actuator and converting a received control command into a signal required to drive the actuator. Therefore, providing a control command to an actuator may include providing the control command to a driver that converts the control command into an appropriate signal to drive the electrical or mechanical device to create the desired movement.
[0074] The robot controller 402 may be implemented by one or more processors, such as a CPU, GPU, and / or other controller of the robot 400. In some implementations, the robot 400 may include a "brain box" that may contain all or part of the functionality of the controller 402. For example, the brain box may provide real-time data bursts to the operating components 403a-403m, each of which includes a set of one or more control commands that specify, among other things, motion parameters of one or more operating components 403a-403m. Although Figure 44, the controller 402 is depicted as an integral part of the robot 400, but in some implementations, all or part of the functionality of the controller 402 may be implemented in a component separate from but in communication with the robot 400. For example, all or part of the functionality of the controller 402 may be implemented on one or more computing devices in wired and / or wireless communication with the robot 400.
[0075] The robot 400 and the general purpose computer assembly 430 communicate via a wired or wireless communication link.
[0076] The computer component 430 includes a proximal computing device 410 and a remote computing device 420 .
[0077] The local computing device 410 and the remote computing device 420 communicate via a wired or wireless communication link.
[0078] The proximal computing device 410 may be a personal computer, a tablet computer, and / or a mobile computing device, such as a smartphone.
[0079] The remote computing device 420 may be a server, a computer cluster, or a supercomputer.
[0080] As a computer program, the introductory language engine 411 can be deployed on the proximal computing device 410. Specifically, the introductory language engine 411 can be a mobile application APP installed on a tablet computer or a smart phone.
[0081] The storage module 412 of the proximal computing device 410 stores information such as the environmental information of the sensor, the robot's long- and short-range memory information, and the guidance language paradigm database.
[0082] As a computer program, the intelligent agent module 421 can be deployed on the remote computing device 420. Since the core program of the intelligent agent module 421 is a large multimodal language model, it is required to be deployed on a computing device with powerful GPU resources, such as a computer cluster or a supercomputer.
[0083] It should be noted that the intelligent agent module 421 can also be deployed on the proximal computing device 410. In this scenario, the intelligent agent module 421 may adopt a lightweight reasoning model, so that the proximal computing device can realize all the reasoning functions of the guide language engine and the intelligent agent.
Claims
1. A method for integrating autonomous execution and interactive execution of an intelligent agent robot, characterized in that The following steps are involved: Step A: Based on the triggering of external environmental factors, the intelligent agent robot generates autonomous guidance words and user guidance words; Step B: The user makes a comprehensive decision based on the user guidance and environmental information and freely chooses whether to issue a high-priority command; Step C: The intelligent agent module asynchronously processes the autonomous guidance language and the high-priority commands from the user to generate an updated subtask sequence described in natural language; Step D: The intelligent agent module converts the latest subtask sequence into code, API sequence or call sequence of different sub-models in real time, collectively referred to as script; Step E: The intelligent agent module executes the script line by line, and transmits the location information of the executed script and the corresponding subtask information back to the memory in real time for the intelligent agent module to call, thereby affecting the subsequent reasoning of the intelligent agent module; Step F: The most fine-grained part of the script that needs to be executed by the robot hardware is sent to the robot hardware for execution by the intelligent agent module through a specific message format; Step G: The robot hardware receives and executes specific action instructions, and then transmits the hardware execution results back to the memory through a specific message format for the intelligent agent module to call, thereby affecting the subsequent reasoning of the intelligent agent module.
2. The interactive execution method according to claim 1, characterized in that: The intelligent agent robot in step A generates autonomous guidance words and user guidance words, including: Step A1: The robot receives its own sensor data in real time and generates multimodal data in a specified format after a specific data processing process; Step A2: The robot's guidance engine module will autonomously determine whether to trigger a new event based on the received multimodal data; Step A3: Once a new event is determined to be triggered, the guide engine module will call and integrate the current environmental information, context data and guide database; Step A4: The guidance language engine module generates an autonomous guidance language for the robot itself, which is subsequently directly input into the intelligent agent module of the robot to generate preliminary autonomous task commands; Step A5: The guidance language engine module generates a user guidance language for the user, which is used to guide the user to participate in the top-level decision-making of the task.
3. The interactive method according to claim 1, characterized in that: The first implementation method of the intelligent agent module in step C asynchronously processing information to generate a subtask sequence includes: Step C11: After receiving the autonomous guidance, the intelligent agent module will combine the long-term and short-term memory data in the memory to perform reasoning and output the corresponding autonomous command; Step C12: the intelligent agent module further decomposes the autonomous command into multiple subtasks described in natural language; Step C13: When receiving a high-priority command from the user, the intelligent agent module will directly generate a subtask sequence described in natural language by combining the long-term and short-term memory data in the memory; Step C14: Optionally, upon receiving a high-priority command from the user, the intelligent agent module first generates a new command that incorporates high- and low-priority information, and then further decomposes the new command into a sub-task sequence.
4. The interactive method according to claim 1, characterized in that: The second implementation method of the intelligent agent module in step C asynchronously processing information to generate a subtask sequence includes: Step C21: After receiving a high-priority command from the user, the intelligent agent module will directly generate a subtask sequence described in natural language by combining the long-term and short-term memory data in the memory; Step C22: After receiving the autonomous guidance words again, the intelligent agent module will combine the long-term and short-term memory data in the memory to perform reasoning and output the corresponding autonomous command; Step C23: The intelligent agent module generates an updated subtask sequence by integrating the subtask sequence of C21 and the autonomous command of C22.
5. The interactive method according to claim 1, characterized in that: The third implementation method of the intelligent agent module in step C asynchronously processing information to generate a subtask sequence includes: Step C31: After receiving the autonomous guidance, the intelligent agent module will combine the long-term and short-term memory data in the memory to perform reasoning and output the corresponding autonomous command; Step C32: the intelligent agent module further decomposes the autonomous command into multiple subtasks described in natural language; Step C33: After receiving the autonomous guidance words again, the intelligent agent module will combine the long-term and short-term memory data in the memory to perform reasoning and output the corresponding new autonomous command; Step C34: The intelligent agent module generates an updated subtask sequence by integrating the subtask sequence of C32 and the new autonomous command of C33.
6. The interactive method according to claim 1, characterized in that: The fourth implementation method of the intelligent agent module in step C asynchronously processing information to generate a subtask sequence includes: Step C41: After receiving a high-priority command from the user, the intelligent agent module will directly generate a subtask sequence described in natural language by combining the long-term and short-term memory data in the memory; Step C42: When receiving a new high-priority command from the user, the intelligent agent module will combine the long-term and short-term memory data in the memory to update and generate a new subtask sequence.
7. The interactive method according to claim 1, characterized in that: The first implementation method of the script location being executed by the robot in step E and the real-time transmission of the location information back to the memory is: Step E11: The subtask name is written at the beginning of the script function block in the form of a comment; Step E12: Whenever a line in the script is executed, print or record the corresponding script line number and the corresponding subtask name; Step E13: Write the script line number and the corresponding subtask name into the memory for the intelligent agent module to call.
8. The interactive method according to claim 1, characterized in that: The second implementation method of the script location being executed by the robot in step E and the real-time transmission of the location information back to the memory is: Step E21: Whenever a line in the script is executed, print or record the corresponding script and subtask name in a log file; Step E22: The log file is continuously written in the memory for real-time reading by the intelligent agent module.
Citation Information
Patent Citations
Intelligent robot control system, method and device based on artificial intelligence
CN104965426A
Robotic manipulation methods and systems for executing a domain-specific application in an instrumented environment with electronic minimanipulation libraries
CN107343382A
Household intelligent accompanying robot and execution control method thereof
CN112828896A
Multi-state Model for Robot and User Interaction
US20120185090A1
Modular autonomous robot distributed control
US20230173675A1
Cited By
Design method and control system and method for large command and control model of unmanned aerial vehicle
CN119645069A
