Robotic Intelligent Systems Based on Large Language Models

Abstract

This application relates to the field of machine learning technology, and more particularly to a robot intelligent system based on a large language model, comprising a functional area and a memory area. The functional area is responsible for receiving text commands, translating them into executable VLA action sequences, and controlling the robot to complete the corresponding tasks. The memory area is divided into a static memory area and a dynamic memory area. The static memory area stores structured data, such as fixed information like the robot's skill library; the dynamic memory area stores executable scripts and the recognition results of the YOLO image segmentation model, providing dynamic data support for task execution. This application improves the robot's understanding and execution capabilities of tasks through the collaborative work of the functional area and the memory area, enhancing the system's adaptability and stability in complex environments.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a robot intelligent system based on a large language model. Background Technology

[0002] Against the backdrop of the rapid development of robotics and artificial intelligence, the application of intelligent agent technology is becoming increasingly widespread. Among them, large language model intelligent agents, as a cutting-edge technology, have demonstrated unique advantages. Large language model intelligent agents are currently the only intelligent agent technology in the world capable of making decisions based on a broad range of human common sense. This characteristic makes them perform exceptionally well in handling complex and varied tasks, simulating human thinking and decision-making processes, thereby providing a more natural and intelligent interactive experience.

[0003] However, its practical application, especially in the field of robotics, still faces many unresolved challenges. Currently, while some technologies attempt to apply large language models to task decomposition and replanning in robots, most of these solutions remain in the theoretical verification stage, lacking practical product applications and systematic technical solutions. Specifically, there are two main key issues: First, user interaction. Achieving natural and efficient interaction is the primary challenge. Existing technologies struggle to accurately understand users' ambiguous intentions and translate them into executable commands. For example, if a user says "I'm thirsty," the robot often cannot accurately interpret and execute the action of picking up a water cup, resulting in insufficient accuracy and robustness that impacts the user experience. Second, anomaly handling. Anomalies are difficult to avoid when robots perform tasks. Existing technologies lack real-time monitoring and effective response mechanisms, making it difficult to react promptly when anomalies occur, easily leading to task failure. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a robot intelligent system based on a large language model. By working collaboratively with the functional area and the memory area, the robot's ability to understand and execute tasks is improved, and the system's adaptability and stability in complex environments are enhanced.

[0005] This application provides a robot intelligent system based on a large language model, the system comprising: Functional areas and memory areas; The functional area is used to receive text commands in text form; translate the text commands into executable VLA action sequences; and control the robot to complete the tasks corresponding to the VLA action sequences. The memory area includes a static memory area and a dynamic memory area; the static memory area is used to store structured data; the dynamic memory area is used to store executable scripts and YOLO image segmentation model recognition results.

[0006] In one optional implementation, the functional area includes a robot dialogue module, a robot motion planning module, and a robot motion execution module; The robot dialogue module is used to convert user voice input into text commands through a speech-to-text model; and to input the text commands into a first visual language model to generate structured command text. The robot motion planning module is used to receive the structured command text and generate an executable VLA action sequence through a second visual language model; The robot motion execution module is used to execute the VLA motion sequence based on the VLA end-to-end model.

[0007] In an optional implementation, the robot motion execution module is further configured to track the execution progress of the VLA motion sequence in real time using a third visual language model and real-time images; trigger a reset command when an anomaly is detected; and transmit the execution log to the robot motion planning module.

[0008] In one optional implementation, the robot dialogue module adopts the Omni model, which consists of a thinker and a generator; the thinker is either a large language model or a visual language model; and the generator is a text-to-speech model.

[0009] In one optional implementation, the robot motion planning module employs either a task planner based on a large language model or a multimodal task planner based on a visual language model.

[0010] In one optional implementation, the robot motion execution module adopts either an integrated VLA model or a heterogeneous architecture vision-language-motion model group; the heterogeneous architecture vision-language-motion model group includes two sub-modules: a slow model and a fast model. The slow model adopts either a visual language model or a large language model, and the fast model is built based on a lightweight real-time control network.

[0011] In an alternative implementation, the system further includes a learning area for self-improvement of the machine learning model already deployed on the robot.

[0012] In one optional implementation, the learning area includes an environmental prediction model improvement module, a large language model improvement module, and a computing resource management module. The environmental prediction model improvement module is used to collect state-action-actual state triplet data in real-time after robot deployment; construct a training sample set based on the triplet data; and iteratively update the model parameters of the environmental prediction model by minimizing the loss function between the predicted state quantity and the actual state quantity. The large language model improvement module is used to collect dialogue interaction data between the robot and the user and the actual execution effect data of the large language model's action planning; generate a scoring signal based on the dialogue interaction data and the actual execution effect data; and select a reinforcement learning strategy through the scoring signal to optimize the large language model's dialogue generation strategy and action planning capabilities. The computing resource management module is used to allocate independent computing resources to the environmental prediction model improvement module and the large language model improvement module; the environmental prediction model improvement module is an improved environmental prediction model, and the large language model improvement module is an improved large language model.

[0013] In one optional implementation, the RLHF algorithm is used as a reinforcement learning strategy when the rating signal originates from a human user, and the RLAIF algorithm is used as a reinforcement learning strategy when the rating signal originates from a machine evaluation.

[0014] In summary, the robot intelligence system based on a large language model provided in this application has at least one of the following beneficial effects: 1. The robot's intelligent system is equipped with a functional area that can receive text commands and translate them into executable VLA (Variable Action Language) sequences. This allows the robot to complete the task corresponding to the VLA sequence. When a user issues a text command with a vague intent (such as "I'm thirsty"), the functional area can receive the command and, through its own translation mechanism, convert this vague intent into a specific, executable VLA sequence (such as moving to the location of the water cup, picking up the water cup, etc.). This controls the robot to complete the corresponding task, solving the problem of existing technologies struggling to accurately understand and translate vague user intents into executable instructions. This improves the accuracy and robustness of the interaction, providing a more natural and efficient interactive experience. 2. The robot's intelligent system is equipped with a memory area, comprising a static memory area and a dynamic memory area. The dynamic memory area stores executable scripts and YOLO image segmentation model recognition results. The YOLO image segmentation model recognition results stored in the dynamic memory area can be used to monitor environmental information during the robot's task execution in real time. For example, during task execution, the YOLO image segmentation model can identify obstacles, hazardous materials, and other situations in the surrounding environment in real time and store the recognition results in the dynamic memory area. When an abnormal situation is detected, the system can quickly react based on the executable scripts stored in the dynamic memory area, adjusting the robot's execution strategy to avoid task failure or property damage caused by abnormal situations, thus solving the problem of existing technologies lacking real-time monitoring and effective response mechanisms. Attached Figure Description

[0015] Figure 1This is a schematic diagram illustrating the architecture of a robot intelligent system based on a large language model, as shown in an embodiment of this application. Figure 2 This is a schematic diagram of a functional area module architecture shown in an embodiment of this application. Detailed Implementation

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] The following will clearly and completely describe the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this invention can be combined interactively without contradicting each other.

[0018] Reference Figure 1 The diagram shown is an architectural schematic of a robot intelligent system based on a large language model, according to an embodiment of this application. The robot intelligent system based on a large language model includes at least two core modules: a functional area and a memory area.

[0019] The functional area is used to receive text commands in text form; translate the text commands into executable VLA action sequences, and control the robot to complete the tasks corresponding to the VLA action sequences.

[0020] The functional areas must at least support three functions: dialogue, thinking, and execution. (Refer to the above.) Figure 2 The functional area includes at least a robot dialogue module (also known as a real-time dialogue system), a robot motion planning module (also known as a robot decision planning system), and a robot motion execution module.

[0021] The robot dialogue module adopts the Omni model, which can be any combination of the following: 1. It consists of a thinker or a generator; 2. The thinker is LLM or VLM, supporting streaming text output; 3. The generator is a TTS model that supports incomplete text as input and real-time generated audio tokens as output.

[0022] Example 1: The thinker in the Omni model can choose the open-source Qwen-Omni-7b Large Language Model (LLM), which supports streaming text output and can generate text responses in real time based on the input dialogue context. The generator in the Omni model integrates the open-source CosyVoice Text-to-Speech (TTS) model, which supports inputting incomplete text (such as sentences or phrases) and outputting natural speech through real-time generated audio tokens. The specific workflow is as follows: the user's speech is converted to text by an STT model (such as Whisper) and then input into the Qwen-Omni-7b thinker; the thinker generates dialogue text and streams it to the CosyVoice generator; CosyVoice converts the text into speech in real time, completing the voice dialogue interaction.

[0023] Example 2: The Omni model can also be trained based on a self-trained LLM and the open-source TTS model CosyVoicez. The LLM and CosyVoice are trained independently and do not interfere with each other. The thinker uses a self-trained domain-adapted large language model (LLM), which optimizes dialogue generation capabilities through fine-tuning and also supports streaming text output. The generator reuses the open-source CosyVoice model, maintaining real-time speech generation capabilities for incomplete text. Its specific workflow is as follows: After the user's speech is converted to text, it is input into a custom LLM. During inference, the LLM outputs tokens one by one. When the output tokens form a sentence, the sentence is immediately sent to the CosyVoice model; CosyVoice generates speech block by block, thus obtaining streaming audio output and ensuring smooth dialogue.

[0024] Example 3: The thinker uses a Visual Language Model (VLM) instead of an LLM. This model can accept multimodal input (such as text + image) and supports streaming text output; the generator remains the CosyVoice model. The specific workflow is as follows: the user's voice and the current environment image are input into the VLM thinker; the VLM generates dialogue text combined with visual information (such as "Do you need to take the water glass in front of you?"); the text is converted into speech output by CosyVoice.

[0025] The robot motion planning module, also known as the task planner, receives confirmation command text and optional environmental images from the robot dialogue module, generates a command sequence conforming to the VLA control protocol, and transmits it to the robot motion execution module. If the initial execution fails, the command sequence is regenerated based on the execution log and environmental images. The robot motion planning module is implemented using any of the following technical solutions: Example 1: Task planner based on open source LLM.

[0026] The robot motion planning module can use an open-source Large Language Model (LLM) as the thinker. The LLM is selected from Llama 3, Falcon, or an improved model thereof, and can directly receive text commands without additional training. The task planner decomposes the input text command (such as "take the water glass from the table to the kitchen") into a step-by-step visual-language-action (VLA) sequence (also called a VLA instruction sequence, or basic action sequence) using preset prompts. Each "basic action" corresponds to a text description, and the VLA action sequence includes at least one of the following: Action type (move / grab / place); Target object description (identified via LLM generalization or specified by the user); Spatial coordinate parameters (based on the environment map or relative position).

[0027] When the precision of instructions generated by LLM does not meet the requirements, the instruction decomposition logic can be optimized through fine-tuning.

[0028] The specific workflow is as follows: the user's voice is converted into a text command (such as "take the water glass from the table to the kitchen") by the STT model; the LLM decomposes the command into step-by-step VLA instructions according to the preset prompt, namely instruction 1: move to the table coordinates (X,Y,Z); instruction 2: identify and grab the "water glass"; instruction 3: move to the target coordinates (X',Y',Z') in the kitchen; if the open source LLM is not effective, the instruction decomposition logic can be optimized by fine-tuning.

[0029] Example 2: A VLM-based multimodal task planner.

[0030] The robot motion planning module uses a Visual Language Model (VLM) as its core thinker. The VLM is selected from Qwen-VL, InternVL, or an improved model thereof, and supports simultaneous processing of text commands and environmental image input. The task planner generates precise instructions through the multimodal understanding capabilities of the VLM. These precise instructions include at least one of the following: Visual characteristics of the target object (color / shape / texture); Spatial relationship description ("the blue water glass on the table"); Dynamic obstacle avoidance path planning; The precise instructions are converted into an executable VLA action sequence using a preset template.

[0031] Its specific workflow is as follows: user commands and the current environment image are input into VLM; VLM combines visual information to generate more precise instructions (such as "grab the blue water glass on the desktop" instead of a generalized instruction); the instructions are converted into VLA execution sequences through preset templates.

[0032] The robot motion execution module receives an image plus robot state data, or an image plus a VLA motion sequence (from the task planner), and generates control signals conforming to the robot controller protocol. The robot motion execution module is implemented using any of the following technical solutions: Example 1: Execution scheme based on integrated VLA model.

[0033] The robot motion execution module can employ an integrated vision-language-motion (VLA) model (such as π_0.5 or its improved version) as the core controller. This integrated VLA model supports multimodal input, including environmental images and robot state data (joint angles, velocities, and end-effector poses). The integrated VLA model directly generates low-level control commands, which include at least one of the following: Joint space control commands (such as "Adjust the angle of left arm joint 1 to 30°"); Cartesian space control commands (such as "the end effector moves to coordinates (X,Y,Z)"); Dynamic force control parameters (such as gripping force threshold); The control commands are executed in real time through the robot's underlying control system (such as ROS or EtherCAT), with an execution frequency of no less than 50Hz.

[0034] Its specific workflow is as follows: the robot acquires environmental images through a camera, and the integrated VLA model directly outputs control commands (such as "adjust the angle of left arm joint 1 to 30°"), which are then executed by the robot's underlying controller (such as ROS).

[0035] Example 2: Execution scheme based on heterogeneous frequency VLA model.

[0036] The robot motion execution module adopts a heterogeneous architecture vision-language-motion model group, which includes two sub-modules: a slow model and a fast model.

[0037] Slow Model: Based on Visual Language Model (VLM) or Large Language Model (LLM), it receives environmental images and text instructions input by the user (such as "grab the water cup"). The VLM or LLM translates the text commands into executable "basic action sequences" as the language modality input of the VLA, so that the VLA model outputs a semantic feature vector, wherein the semantic features include at least one of the following: Spatial positioning of the target object (e.g., "the water glass is located on the left side of the table"); Action type identifier (move / grab / place); Environmental constraints (obstacle avoidance zone markings).

[0038] Fast model: Built on a lightweight real-time control network (such as the Figure Helix architecture), it receives semantic features output by the slow model and generates high-frequency control signals, wherein the control signals include at least one of the following: Joint torque / speed command (execution frequency ≥ 200Hz); Impedance control parameters (stiffness / damping coefficient); Emergency stop triggering conditions.

[0039] The fast model needs to be specifically trained using robot body data (such as joint trajectories and force sensor data) to adapt to the dynamic characteristics of specific mechanical structures.

[0040] Its specific workflow is as follows: receiving image and text instructions (such as "grab the water cup"); outputting semantic features (such as "the water cup is on the left side of the table"); calculating joint torque and speed in real time based on semantic features; and outputting high-frequency control signals (200Hz).

[0041] To facilitate understanding of the inventive concept of this application, the following provides a workflow example where the task planner uses VLM and the action execution module uses a different frequency VLA model. VLM decomposes user commands into multi-step instructions (such as "move → grab → place"). Each step of the instruction triggers the slow model of the different frequency VLA model to generate semantic features. The fast model adjusts the control quantity in real time based on the features to avoid the generalization problem of the integrated model.

[0042] The memory area includes a static memory area and a dynamic memory area; the static memory area is used to store structured data; the dynamic memory area is used to store executable scripts and YOLO image segmentation model recognition results.

[0043] In this embodiment, the static memory area serves as the robot's long-term knowledge base, storing the following structured data: Semantic map: Records environmental spatial information (such as room layout and furniture location) in the form of a topological map, supporting path planning and spatial reasoning.

[0044] Voiceprint memory: Stores the user's voiceprint feature vector for identity recognition and personalized response during voice interaction (such as "Welcome back, Mr. Zhang").

[0045] Dialogue history: Stores user-bot dialogue text in chronological order, supporting contextual understanding and multi-turn dialogue management.

[0046] Action library: Defines the set of actions that the robot can perform (such as "grabbing a water cup" or "opening a door"). Each action is associated with pre-trained parameters (such as robotic arm joint angles and movement speed) and triggering conditions (such as object distance < 0.5m). For example, when a user issues the instruction "hand me the water glass on the table", the system retrieves from the static memory area: the coordinates of the "table" in the semantic map; the contextual association of the "water glass" in the dialogue history (as mentioned earlier, "hot water"); and the action parameters and dependent components (robotic arm, vision module) of "grabbing the water glass" in the action library.

[0047] The dynamic memory area stores executable scripts, supporting real-time task decomposition and subtask scheduling. Scripts are defined in Python or JSON format and contain the following elements: Triggering conditions: based on static memory area data or sensor input (e.g., "when a 'water cup' is detected in the semantic map and the user's location is within 1 meter").

[0048] Execution flow: Call the atomic action combination in the action library (such as "move to the table → locate the water cup → grab → deliver").

[0049] Exception handling: Define the rollback strategy when the script is interrupted (such as replanning the path or requesting user assistance when the fetching fails).

[0050] The specific script execution flow is as follows: The YOLO model performs real-time segmentation on the camera input image, outputting object categories and bounding boxes (e.g., "water cup: [x1,y1,x2,y2]"). The script in the dynamic memory area checks whether the YOLO detection result meets the triggering conditions (e.g., whether the "water cup" is within the field of view and operable). If the conditions are met, the script calls the "grab water cup" action from the action library and generates multimodal large model prompts (e.g., "The water cup was detected 0.3 meters to the right; it is recommended to grab it with your right hand"). The action execution result (success / failure) is returned to the dynamic memory area, the script state is updated, and the dialogue history is recorded in the static memory area.

[0051] The execution of the dynamic memory script is highly dependent on the accuracy of environmental perception. To improve the reliability of task planning, the system implements a collaborative optimization strategy at the visual perception and multimodal inference levels. In visual perception, the YOLO model is configured differently: if the scene only needs to recognize 80 common object categories supported by open-source YOLOv8 (such as "water cup" and "chair"), the system adopts a lightweight deployment strategy, directly loading pre-trained weights to reduce computational overhead; if it needs to recognize custom objects (such as "special-type tools"), the model is fine-tuned based on labeled data to optimize the mean average precision (mAP) to ≥95%, ensuring that the detection results meet the action triggering conditions (such as "water cup" confidence > 0.9). At the multimodal inference level, the system encodes the YOLO detection results (object category, location, confidence) into structured text (such as...) through prompt word engineering. <object>water cup <confidence> 0.92 <position>On the right side, the input is used to perform task reasoning on a large model; simultaneously, it combines the dialogue history in the static memory area (such as the user mentioning "hot water" earlier) with voiceprint memory (to recognize user emotions) to generate a more accurate response (such as "I detected your usual blue water cup, shall I hand it to you now?"). For example, when the user says "Get me the water cup," the dynamic memory area activates the grasping script, YOLO detects the position of the "water cup" in the image and outputs the bounding box coordinates, the script verifies whether the "water cup" is within the effective range of the "grab" action in the action library (distance < 1m), if the condition is met, the robotic arm is invoked to grasp it, and the execution result (such as "grab successful") is fed back to the static memory area to update the dialogue history, thus forming a complete closed loop from perception to decision-making to execution.

[0052] In some embodiments, to address uncertainties in complex environments and ensure system robustness, the system constructs a multi-level exception handling and dynamic rollback mechanism in the interaction between the static and dynamic memory areas. First, proactive defense is achieved through user feedback monitoring: the static memory area analyzes negative keywords (such as "error" and "retry") in the dialogue history in real time. If their frequency suddenly increases by 20%, the dynamic memory area is triggered to roll back to the previous stable script version. Simultaneously, voiceprint memory is used to identify user emotions (such as anger), prioritizing complaint-type commands to improve user experience. Second, a task failure rate threshold is set as dynamic protection: the dynamic memory area counts the number of consecutive failures of key actions (such as "grab a water cup"). If the number exceeds 3 or the failure rate increases by 2 times, the script is immediately paused and an alarm is triggered. Simultaneously, the semantic map is updated with environmental obstacle information to replan the path. Furthermore, the system implements resource protection strategies to ensure core function stability: when a script in the dynamic memory area consumes more than 90% of GPU memory or causes a main task delay exceeding 500ms, low-priority scripts (such as non-emergency inspection tasks) are forcibly terminated, and the static memory area backs up the current state, providing a rapid recovery basis for subsequent rollbacks. For example, when the YOLO model mistakenly detects a "water cup" as a "vase" due to light interference, causing the robotic arm to fail to grasp it, the dynamic memory area records that after two consecutive failures, the script is paused and switched to manual confirmation mode. The static memory area simultaneously updates the dialogue history to "Water cup not found, please describe the location again." At the same time, the confidence threshold of the corresponding category of the YOLO model is lowered to optimize subsequent detection, forming a complete closed loop from anomaly perception to policy response to state repair.

[0053] In an alternative implementation, the robot intelligence system may further include a learning area for self-improvement of the robot's deployed machine learning model.

[0054] In this embodiment, the learning area is responsible for implementing the self-improvement mechanism of the machine learning model already deployed in the actual application of the robot. The learning area includes an environment prediction model improvement module, a large language model improvement module, and a computing resource management module.

[0055] The environmental prediction model improvement module is used to collect state-action-actual state triplet data in real-time after robot deployment; construct a training sample set based on the triplet data; and iteratively update the model parameters of the environmental prediction model by minimizing the loss function between the predicted state quantity and the actual state quantity.

[0056] For environmental prediction models (such as wind prediction (drones), wave surge prediction (unmanned surface vessels), and target trajectory prediction (all types of robots)), real-time state-action-actual state triplet data (s, ...) are collected during robot execution. a , ) Conduct continuous training to meet the requirements Where s represents the robot's state variables at the current moment (including joint angles, velocity, and end-effector pose). a This refers to the control quantity (joint torque / speed command) at the current moment. This refers to the state variable after a specified time interval. This type of environmental prediction model extracts a portion of the robot's own trajectory D (which may come from a real or simulated environment) during training, and then minimizes the prediction error. The optimization objective is to iteratively update its own internal trainable parameters. To execute control quantity a The actual observed state quantities within the subsequent Δt time interval. Clearly, the robot will also generate a series of realistic motion trajectories when it is working normally after deployment, which can be used to further train similar models.

[0057] The large language model improvement module is used to collect dialogue interaction data between the robot and the user, as well as actual execution effect data of the large language model's action planning; generate a scoring signal based on the dialogue interaction data and the actual execution effect data, and select a reinforcement learning strategy through the scoring signal to optimize the large language model's dialogue generation strategy and action planning capabilities.

[0058] For Large Language Models (LLMs), the dialogue interactions between the LLM and users, and the actual execution effects of the LLM's action planning, are all important corpora. These can be further improved using RLHF or RLAIF algorithms as reinforcement learning strategies. RLAIF and RLHF are essentially the same type of reinforcement learning algorithm for LLMs. The input to these algorithms consists of LLM dialogue responses and a rating (e.g., satisfied or dissatisfied, partial order of dialogue satisfaction). When this rating signal comes from a human, the RLHF (Human Feedback) algorithm is used as the reinforcement learning strategy; if the rating signal comes from a machine, the RLAIF (Machine Feedback) algorithm is used. When both types of feedback exist simultaneously, the final rating is synthesized according to preset weights (e.g., human feedback weight 0.7, machine feedback weight 0.3). The optimal combination of weights is determined through offline experiments. An independent reinforcement learning training process is established within the learning area, including the following core components: Reward model: It adopts the Transformer architecture. The input is the concatenated text of the dialogue history and the model's response. The output is a scalar reward value. The training objective is to minimize the mean squared error between the predicted reward and the actual score. Policy Network: Consistent with the structure of the large language model to be optimized, the generated policy is adjusted according to the reward signal through the PPO algorithm, and KL divergence constraint (coefficient β=0.01) is introduced during training to prevent policy drift; Experience replay pool: Stores the most recent 100,000 dialogue-rating-action triples, and samples them according to priority (prioritizing samples with high reward differences) to construct training batches; Distributed training framework: The policy network and reward model are deployed on independent GPU nodes, gradient synchronization is achieved through the NCCL communication library, the batch size is set to 256, and the learning rate adopts a linear warm-up + cosine decay strategy.

[0059] In addition to self-improvement of environmental prediction models and large language models, for end-to-end models with action planning capabilities (such as multimodal large models combining vision, language, and action), the learning area employs an asynchronous optimization mechanism for self-improvement, specifically including: Data isolation: Interaction data generated during the execution of the main task is directly written to the learning area cache using zero-copy technology to avoid the main thread waiting; Periodic fine-tuning: A model update is triggered every 1000 complete execution trajectories (including state observations, action outputs, and environmental rewards). During the update, the encoder parameters are frozen, and only the decision head network is optimized. Security Validation: The optimized model must pass a security sandbox test. Validation metrics include: instruction compliance (proportion of rejected dangerous instructions ≥ 99.9%), task success rate (≥ 5% improvement compared to the baseline model), and response latency (P99 latency ≤ 500ms). If any metric fails to meet the standard, the model will automatically roll back to the previous version.

[0060] The computing resource management module is used to allocate independent computing resources to the environmental prediction model improvement module and the large language model improvement module; the environmental prediction model improvement module is an improved environmental prediction model, and the large language model improvement module is an improved large language model.

[0061] Meanwhile, the learning area achieves efficient resource utilization through the following strategies, allocating independent computing resources to the environmental prediction model improvement module and the large language model improvement module: Priority queue: Allocate computing resources based on model type (large language model > environment prediction model > end-to-end model) and task urgency (real-time feedback optimization > offline fine-tuning); Elastic scaling: When the system load is below 30%, automatically allocate additional GPU resources for RLHF training; when the main task latency exceeds the threshold, pause all non-critical optimization tasks. Energy efficiency optimization: NVIDIA MIG technology is used to divide the A100 GPU into 7 independent instances, with different instances used for different model training tasks to avoid resource contention; Monitoring and Alerts: Real-time tracking of metrics such as GPU utilization, memory usage, and training steps in the learning area. When resource utilization exceeds 95% for 5 consecutive minutes, a capacity expansion warning is triggered.

[0062] Furthermore, all optimized model versions (i.e., improved models) must be managed through a standardized process. Specifically, firstly, metadata is recorded, fully preserving core information such as the model version number, training data hash value, key performance indicators (e.g., BLEU score, task success rate), and dependent component versions. Then, the canary release phase begins, deploying the new version to 5% of the robot cluster for A / B testing, focusing on comparing the baseline model's performance in key metrics such as user satisfaction and task completion rate. If, during testing, an anomaly occurs such as a sudden increase in user complaint rate exceeding 20%, a critical task failure rate more than doubling, or system resource exhaustion leading to main task interruption, the system will automatically trigger a rollback mechanism, immediately switching back to the previous stable version. Only after all A / B test metrics meet the standards can the full release process be initiated, gradually expanding the deployment to 100% of the cluster, with the entire process required to take no more than 72 hours to ensure controllable model iteration and traceable risks.

[0063] In some embodiments, the robot intelligent system may also include other models, such as navigation and dexterous hand control, typically implemented based on traditional technologies such as ROS. Taking the navigation model as an example, it can combine SLAM (Simultaneous Localization and Mapping) algorithms to build an environmental map using sensor data and locate the robot's position in real time, planning safe and efficient movement paths for the robot. The dexterous hand control model can precisely control the joint movements of the dexterous hand according to task requirements, enabling actions such as grasping and manipulating different objects. The intelligent agent based on the large language model interacts and collaborates through the communication mechanism provided by ROS, jointly improving the overall performance of the robot intelligent system and enabling it to complete various complex tasks more comprehensively and stably.

[0064] To address the shortcomings of existing robot systems in terms of stability, security, and computational resource allocation, and to verify the robot intelligent system proposed in this application, the following approach utilizes four collaborative intelligent agents, combined with the resource scheduling capabilities of functional and memory areas, to achieve end-to-end optimization of task planning, execution monitoring, and user interaction, significantly improving task success rate and hardware utilization in complex scenarios. Specifically, through progress tracking and replanning mechanisms, the success rate of a single task is increased from 50% in existing solutions to over 85%; by employing model reuse and fine-tuning strategies, the memory usage is reduced from over 100GB in traditional solutions to a single 80GB A100 graphics card; and through log recording and voice feedback, users and the technical team can quickly pinpoint the cause of failure.

[0065] The following provides a robot intelligent system consisting of four intelligent agents based on a large model, whose division of labor and interaction logic are shown in Table 1.

[0066] Table 1:

[0067] For dialogue agents, natural interaction is achieved through a voice-text-voice closed loop. Specifically, this includes the following steps: Step 1, Speech-to-Text (STT): The Whisper model (7 parameters) is used to transcribe the user's speech in real time, supporting mixed Chinese and English input; Step 2, Text Response Generation: A general-purpose VLM (such as LLaVA-7B) receives the transcribed text and the current image, and generates dialogue response text; Step 3, Text-to-Speech (TTS): The VITS model is used to convert the reply text into natural speech, supporting emotion adjustment (such as friendly, serious).

[0068] For example, user: "Hello, I'm a little thirsty." → STT to text → VLM to analyze image (identify kitchen location) → Generate reply: "Would you like me to get you a water glass from the kitchen?" → TTS speech output.

[0069] For an intent-transforming agent, the mapping of fuzzy to explicit instructions is achieved, specifically including the following steps: Step 1, Input Processing: Combine short-term dialogue history (such as the previous 3 rounds of dialogue) with the current image to understand the user's implicit needs; Step 2, Instruction Generation: The general VLM converts the fuzzy expression into structured commands and marks it as "requires confirmation"; Step 3: User Confirmation: The user confirms the command via voice through the dialogue agent and marks it as "Confirmed".

[0070] For example, user: "I'm so thirsty" + image (there's a water glass in the kitchen) → VLM generates command: "Get a water glass from the kitchen for the user [confirmation required]" → the dialogue agent broadcasts → user: "Okay" → marked as "confirmed".

[0071] For action planning agents, task decomposition and dynamic replanning are implemented, specifically including the following steps: Step 1, Initial Planning: A dedicated VLM (after fine-tuning) receives confirmation commands and images, and outputs an executable action sequence (such as "move → grab → deliver"). Step 2, Failure Handling: If the execution fails, the planning agent analyzes the execution log (e.g., "Grab failed: object slipped"), adjusts the action parameters (e.g., increases the grasping force) or replaces the action (e.g., uses both hands to grasp). Step 3, Replanning Verification: Verify the replanning scheme through a simulation environment to ensure that the same error is not repeated.

[0072] For example, the command is: "Get a water cup from the kitchen" → Initial planning: "Move to the kitchen → Grab the water cup → Deliver to the user" → Grab failure log → Replanning: "Move to the kitchen → Adjust the robotic arm angle → Grab the water cup → Deliver" → Execution successful.

[0073] For the execution monitoring agent, progress tracking and anomaly reset are implemented, specifically including the following steps: Step 1, Progress Judgment: A dedicated VLM analyzes the real-time image every 2 seconds and matches the action progress template (e.g., "Grabbing in progress: distance between finger and object < 5cm"). Step 2, Anomaly Detection: If the progress stops or deviates from the template (e.g., "Moving: Robot position not updated"), immediately trigger a reset command; Step 3, Safety Protection: When resetting, stop all actions first, retreat to a safe position, and then replan.

[0074] For example, the process is as follows: "Grab a water cup" → VLM detection: "Finger distance to object > 10cm (abnormal)" → Trigger reset → Robot stops moving → Reverses to initial position → Re-plans task.

[0075] It should be noted that the reset command refers to a highly reliable combination of actions used to restore the robot system from an abnormal state or task interruption state to an initial safe state. Its specific implementation methods include, but are not limited to, the following two types: Automatic reset: A sequence of actions automatically triggered and executed by the robot's built-in control algorithm. The control algorithm includes, but is not limited to, any one or a combination of joint space trajectory planning algorithms based on position feedback, impedance control algorithms based on force / torque control, and relocation algorithms based on vision / laser positioning. This application does not limit the specific form of the automatic reset algorithm, and any control method that can realize the recovery of the robot's state falls within the protection scope of this application.

[0076] Manual reset: A manual intervention reset method triggered by an external operating terminal (such as a remote control, touch screen, voice command device) or physical switch, including but not limited to any one or combination of remotely sending a reset command, manually switching to a safety mode, and resetting via a physical emergency stop button; the manual reset method and its triggering mechanism are both within the scope of protection of this application.

[0077] The dialogue agent and intent translation agent share a general-purpose VLM (such as LLaVA-7B) responsible for dialogue and intent understanding, requiring no fine-tuning. The action planning agent and execution monitoring agent use a dedicated VLM (such as Qwen-VL-7B), which, after fine-tuning, supports action planning and monitoring, and the output format needs to be adjusted according to the robot's list of executable actions. The following table 2 provides an application configuration example of a Visual Language Model (VLM) in a robot task processing system based on multiple agents and shared memory.

[0078] Table 2:

[0079] As can be seen, a single A100 graphics card supports the entire process, with a video memory usage of approximately 60GB (including buffer). If VLM technology advances to support the entire process with a single model in the future, this could be compressed to 32GB for consumer-grade graphics cards (such as the RTX 4090).

[0080] The robot intelligence system based on a large language model proposed in this application is applicable to all autonomous intelligent robots equipped with high-performance computing platforms. High-performance computing platforms are generally divided into two categories: one is the robot's edge computing platform, such as the Jetson development board; the other is a remote computing platform, such as a remote server equipped with high-performance GPUs like the A100. When the high-performance computing platform is remote, the robot communicates with the computing platform via a network. To facilitate understanding of the inventive concept of this application, the following is an example of a robot task execution flow based on a remote or edge high-performance computing platform: Step 1: The user has a voice conversation with the robot. For example, the user says, "I'm so thirsty."

[0081] Step 2: The robot analyzes the user's actual needs and obtains a clear command through "dialogue history + image → command text". For example, "fetch water from the kitchen", while the "confirm?" flag is set to "no".

[0082] Step 3: The robot requests confirmation of the parsed command from the user through voice interaction. For example, it might ask, "Do you want me to get a bottle of water from the kitchen?"

[0083] Step 4: If user confirmation is received, proceed to Step 5; otherwise, return to Step 1 and wait for new voice input from the user. This decision is also made by the agent "dialogue history + image → command text". Because the user may add new information during confirmation, the confirmation process may involve repeatedly executing the loop of Steps 1-4.

[0084] Step 5: Execute the "Command Text + Image → Execution Plan" agent to obtain the robot's motion plan.

[0085] Step 6: Using the VLA model and SLAM navigation algorithm, sequentially execute the action list from Step 5. Simultaneously, start a timer and run a concurrent process to execute the "execution plan + current image → execution progress" agent at regular intervals. If an abnormal progress is detected, switch to Step 7; otherwise, after all actions are completed, return to Step 1 and wait for new user input.

[0086] Step 7: When an execution anomaly is detected, the robot forcibly executes the "reset" command, and at the same time summarizes and categorizes the reasons for the failure through the large model used for dialogue in the progress report, and broadcasts it to the user via voice.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.< / position> < / confidence> < / object>

Claims

1. A robot intelligent system based on a large language model, characterized in that, The system includes: Functional areas and memory areas; The functional area is used to receive text commands in text form; translate the text commands into executable VLA action sequences; and control the robot to complete the tasks corresponding to the VLA action sequences. The memory area includes a static memory area and a dynamic memory area; the static memory area is used to store structured data; the dynamic memory area is used to store executable scripts and YOLO image segmentation model recognition results.

2. The robot intelligent system based on a large language model according to claim 1, characterized in that, The functional area includes a robot dialogue module, a robot motion planning module, and a robot motion execution module; The robot dialogue module is used to convert user voice input into text commands through a speech-to-text model; and to input the text commands into a first visual language model to generate structured command text. The robot motion planning module is used to receive the structured command text and generate an executable VLA action sequence through a second visual language model; The robot motion execution module is used to execute the VLA motion sequence based on the VLA end-to-end model.

3. The robot intelligent system based on a large language model according to claim 2, characterized in that, The robot motion execution module is also used to track the execution progress of the VLA motion sequence in real time through a third visual language model and real-time images; trigger a reset command when an anomaly is detected, and transmit the execution log to the robot motion planning module.

4. The robot intelligent system based on a large language model according to claim 2, characterized in that, The robot dialogue module adopts the Omni model, which consists of a thinker and a generator; the thinker can be either a large language model or a visual language model; the generator is a text-to-speech model.

5. The robot intelligent system based on a large language model according to claim 2, characterized in that, The robot motion planning module can use either a task planner based on a large language model or a multimodal task planner based on a visual language model.

6. The robot intelligent system based on a large language model according to claim 2, characterized in that, The robot motion execution module adopts either an integrated VLA model or a vision-language-motion model group with a heterogeneous architecture; the vision-language-motion model group with a heterogeneous architecture includes two sub-modules: a slow model and a fast model. The slow model adopts either a visual language model or a large language model, and the fast model is built based on a lightweight real-time control network.

7. The robot intelligent system based on a large language model according to any one of claims 1 to 6, characterized in that, The system also includes a learning area for self-improvement of the machine learning model already deployed on the robot.

8. The robot intelligent system based on a large language model according to claim 7, characterized in that, The learning area includes an environmental prediction model improvement module, a large language model improvement module, and a computing resource management module. The environmental prediction model improvement module is used to collect state-action-actual state triplet data in real-time after robot deployment; construct a training sample set based on the triplet data; and iteratively update the model parameters of the environmental prediction model by minimizing the loss function between the predicted state quantity and the actual state quantity. The large language model improvement module is used to collect dialogue interaction data between the robot and the user and the actual execution effect data of the large language model's action planning; generate a scoring signal based on the dialogue interaction data and the actual execution effect data; and select a reinforcement learning strategy through the scoring signal to optimize the large language model's dialogue generation strategy and action planning capabilities. The computing resource management module is used to allocate independent computing resources to the environmental prediction model improvement module and the large language model improvement module; the environmental prediction model improvement module is an improved environmental prediction model, and the large language model improvement module is an improved large language model.

9. The robot intelligent system based on a large language model according to claim 8, characterized in that, When the rating signal originates from human users, the RLHF algorithm is used as the reinforcement learning strategy; when the rating signal originates from machine evaluation, the RLAIF algorithm is used as the reinforcement learning strategy.

Images