Robot task data processing method based on double-model architecture and robot

Abstract

The invention provides a robot task data processing method based on a double-model architecture and a robot, and belongs to the technical field of electric digital data processing. Generating a task sequence corresponding to the first user instruction through the decision-making layer text generation large model, and generating a first action list of each task in the task sequence through the execution layer multi-mode large model. And the ROS main control unit calls an execution unit to execute the first action list so as to complete the current task, obtain task feedback information and update the context information set. The ROS main control unit enters a preset state, when a preset condition is not met, the execution layer multi-mode large model generates a first action list of each task according to the execution layer multi-mode large model, when the preset condition is met, the ROS main control unit enters a waiting state, and after a temporary instruction is obtained, the execution layer multi-mode large model generates a second action list based on the updated context information set and executes the second action list. The defect that modal interference is prone to occurring in the process of processing complex tasks during single AI large model control can be overcome.

Description

Technical Field

[0001] The present application belongs to the field of electronic digital data processing technology, and specifically relates to a robot task data processing method and a robot based on a dual-model architecture. Background Art

[0002] Currently, the most common control method in the control system of AI large-model robots is to use a multimodal large model + single-shot inference. This single-model architecture usually uses a multimodal large model to simultaneously undertake multiple functions such as semantic understanding, task planning, and motion control. In actual applications, it is widely used in scenarios such as smart home robots and service robots to perform tasks such as home cleaning, item delivery, and customer reception.

[0003] However, this existing single model + single inference approach has many obvious flaws: First, multimodal large models are relatively weak in semantic understanding and logical reasoning. When faced with complex and abstract human instructions, they struggle to accurately parse and rationally plan task steps, resulting in inefficient or even failed task execution. Second, when performing complex tasks, a single model faces the problem of modal interference. Because it needs to simultaneously process natural language, environmental perception information, and motion control instructions, information from different modalities easily interferes with each other, resulting in language ambiguity and incorrect actions, causing confusing action logic that fails to conform to objective facts. Third, the training samples of a single model are relatively complex, and a large number of examples are needed to help the large AI model understand. The model needs to learn the rules of the three parts of "natural language understanding + environmental perception + motion control" in the sample at the same time. The training is difficult and prone to overfitting, which makes the model's generalization ability in actual applications poor and difficult to adapt to the diverse task requirements in different scenarios.

[0004] Fourth, the method of generating an action list through single inference is suitable for simple task scenarios, that is, it assumes that every action of the robot can be executed successfully. However, in actual complex scenarios, the robot may fail when performing a certain action, which will have a chain reaction on the execution of subsequent tasks. This method is equivalent to open-loop control, and when the robot fails to perform an action, it cannot effectively adjust the subsequent behavior of the robot.

[0005] Traditional embodied intelligent robots often use a single model architecture to handle complex tasks, which has problems such as coupling of semantic parsing and motion control, significant modal interference, and high complexity of training samples, resulting in low task execution efficiency and insufficient robustness. Summary of the Invention

[0006] This application provides a robot task data processing method and robot based on a dual-model architecture to address the shortcomings of the current single AI large model control method, which is prone to modal interference and difficult training when processing complex tasks.

[0007] This application provides a robot task data processing method based on a dual-model architecture, which is applied to the decision-making layer text generation model, the execution layer multimodal large model, the ROS master control unit and the execution unit of the robot control system, including: Step a: The ROS main control unit obtains a text recognition result corresponding to a first user instruction, creates a first context information set for a current task cycle, and adds the text recognition result to the first context information set to obtain a second context information set; Step b: the decision-level text generation model outputs a first task sequence according to the text recognition result, and adds the first task sequence to the second context information set to obtain a third context information set; Step c: the execution layer multimodal large model determines a current sub-task from the first task sequence based on the third context information set, generates a first action list corresponding to the current sub-task, and adds the first action list to the third context information set to obtain a fourth context information set; Step d: the ROS master control unit drives the execution unit to execute the first action list, obtains execution status feedback information, and adds the execution status feedback information to the fourth context information set to obtain a fifth context information set; Step e: the execution layer multimodal large model determines whether a preset condition is met based on the execution status feedback information; If not, the fifth context information set is used as the third context information set, and steps c to e are repeated until the current task cycle is completed; If the conditions are met, the system enters a waiting state, and when a temporary instruction is obtained in the waiting state, the execution layer multimodal large model determines a second task sequence according to the fifth context information set and the temporary instruction.

[0008] According to the robot task data processing method based on the dual-model architecture provided by the present application, before the execution layer multimodal large model determines the second task sequence according to the fifth context information set and the temporary instruction, the method also includes: determining whether a second user instruction is obtained in the waiting state; when the second user instruction is obtained, the ROS main control unit determines whether the second user instruction is a temporary instruction; the execution layer multimodal large model determines the second task sequence according to the fifth context information set and the temporary instruction, including: when the second user instruction is a temporary instruction and does not meet the preset conditions, the execution layer multimodal large model parses the temporary task corresponding to the temporary instruction, and generates a second task sequence for the temporary task based on the temporary task and the fifth context information set.

[0009] According to the robot task data processing method based on the dual-model architecture provided in the present application, the execution layer multimodal large model parses the temporary task corresponding to the temporary instruction, and generates a second task sequence of the temporary task based on the temporary task and the fifth context information set, including: parsing the temporary instruction to obtain the temporary task corresponding to the temporary instruction, the temporary task including a task object and task requirements; matching the task object in the fifth context information set to obtain a matching result; determining the current state of the task object based on the matching result; and generating the second task sequence of the temporary task based on the current state, the task requirements and the fifth context information set.

[0010] According to the robot task data processing method based on the dual-model architecture provided in the present application, the second task sequence of the temporary task is generated according to the current state, the task requirements and the fifth context information set, including: determining the task type of the temporary instruction according to the task requirements; determining whether there is a reference instruction with the same task type as the temporary instruction in the first user instruction according to the fifth context information set; if so, obtaining the reference task sequence corresponding to the reference instruction in the fifth context information set; generating the second task sequence of the temporary task according to the current state, the task requirements and the reference task sequence.

[0011] According to the robot task data processing method based on the dual-model architecture provided in the present application, the second task sequence of the temporary task is generated according to the current state, the task requirements and the reference task sequence, including: determining the initial position of the task object according to the current state; determining the target position of the task object according to the task requirements; obtaining a map mapping table of the site where the robot is located; determining the initial mapping area of ​​the initial position of the task object and the target mapping area of ​​the target position according to the map mapping table; obtaining the reference initial mapping area and reference target mapping area of ​​the task object in the reference instruction; replacing the reference initial mapping area of ​​the task object in the reference task sequence with the initial mapping area, and replacing the reference target mapping area with the target mapping area to obtain a second task sequence.

[0012] According to the robot task data processing method based on the dual-model architecture provided in the present application, the method also includes: when the second user instruction is not obtained in the waiting state, or when the second user instruction is not a temporary instruction, or when the temporary instruction indicates that the preset condition is met, calling the completion dialogue function to reset the fifth context information set and determining that the current task cycle is over.

[0013] According to the robot task data processing method based on the dual-model architecture provided in the present application, satisfying the preset conditions includes any one of the following: obtaining a target user instruction, the target user instruction is used to instruct the robot to stop working; the execution status feedback information indicates that the robot has failed the current task for a preset number of consecutive times; the execution layer multimodal large model determines that the current task is an unfinishable task.

[0014] According to the robot task data processing method based on the dual-model architecture provided in the present application, the robot control system also includes a speech recognition model, and the ROS main control unit obtains the text recognition result corresponding to the first user instruction, including: after the ROS main control unit monitors the preset wake-up word, it obtains the user voice information; the speech recognition model determines the first user instruction based on the user voice information uploaded by the ROS main control unit, and generates a text recognition result corresponding to the first user instruction.

[0015] The present application also provides a robot control system, comprising a decision-making layer text generation large model, an execution layer multimodal large model, a ROS master control unit and an execution unit; The decision-making layer text generation large model is used to perform the steps performed by the decision-making layer text generation large model in any of the above methods; The execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any of the above methods; The ROS master control unit is used to execute the steps performed by the ROS master control unit in any of the above methods; The execution unit is used to execute the steps executed by the execution unit in any of the above methods.

[0016] The present application also provides a robot, comprising a decision-making layer text generation large model, an execution layer multimodal large model, a ROS main control unit and an execution unit; The decision-making layer text generation large model is used to perform the steps performed by the decision-making layer text generation large model in any of the above methods; The execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any of the above methods; The ROS master control unit is used to execute the steps performed by the ROS master control unit in any of the above methods; The execution unit is used to execute the steps executed by the execution unit in any of the above methods.

[0017] In the robot task data processing method and robot based on the dual-model architecture provided by the present application, a task sequence corresponding to the first user instruction is generated by a decision-making layer text generation model, and a first action list for each task in the task sequence is generated by an execution layer multimodal large model. The ROS main control unit calls the execution unit to execute the first action list to complete the current task, obtain task feedback information, and update the context information set. When the preset conditions are not met, the first action list for each task generated by the execution layer multimodal large model is used. When the preset conditions are met, the ROS main control unit enters a waiting state, and after obtaining a temporary instruction, the execution layer multimodal large model generates a second action list based on the updated context information set and executes it.

[0018] This application solves the drawbacks of the current single AI large model control method that is prone to modal interference and difficult to train when handling complex tasks by combining dual models with dynamic reasoning. The multimodal large model is inferior to the text generation large model in the ability to understand natural language semantics and logical reasoning, and is prone to modal interference when performing complex tasks. Therefore, the decision logic and action control are decoupled in the system design. The text generation large model focuses on semantic understanding and task planning, avoiding the complexity of the traditional single model that needs to deal with the details of the action at the same time during semantic parsing. The multimodal large model focuses on action generation and environmental interaction, reducing the confusion of action logic caused by the complexity of the task of the single model, which does not conform to objective facts.

[0019] Secondly, the complexity of model training samples is significantly reduced. In single-model reasoning, large multimodal models must simultaneously perform "natural language understanding, environmental perception, and motion control," which can easily lead to modal interference (such as incorrect movements due to language ambiguity). The dual-model approach uses phased processing, allowing the decision layer to focus on language space mapping and the execution layer to focus on visual-motor space mapping, providing diverse training samples and reducing the risk of overfitting.

[0020] In addition, dynamic reasoning can help the robot adjust its actions in real time. After each action is completed, the robot will feedback the status of the action to the multimodal large model, solving the problem of chain reactions to subsequent actions caused by the failure of one action when the robot performs a long sequence of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a schematic diagram of the composition of the robot control system provided by this application.

[0023] Figure 2 It is a flow chart of the robot task data processing method based on the dual-model architecture provided in this application.

[0024] Figure 3 This is a diagram of task sequence generation provided by this application.

[0025] Figure 4 This is a schematic diagram of the execution process of the execution layer multimodal large model provided by this application.

[0026] Figure 5 This is a schematic diagram of the process of adding context information sets provided by this application.

[0027] Figure 6 This is a map mapping diagram provided by this application.

[0028] Figure 7 This is a schematic diagram of the real environment execution provided by this application.

[0029] Figure 8 This is a schematic diagram of resetting the context information set provided by this application.

[0030] Figure 9 This is a full-process diagram of the robot task data processing method based on the dual-model architecture provided in this application.

[0031] Figure 10 This is a single-task cycle flow chart of the robot task data processing method based on the dual-model architecture provided in this application. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0033] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0034] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] Currently, the most common control approach for large-scale AI robot models is to use a multimodal large model combined with a single inference approach. This single-model architecture typically employs a large multimodal model to simultaneously handle multiple functions, including semantic understanding, task planning, and motion control. This can lead to modal interference and increased training difficulty when handling complex tasks.

[0036] To address the above issues, the present invention provides a robot task data processing method and a robot based on a dual-model architecture. The present invention is described in detail below with reference to the accompanying drawings.

[0037] See also Figure 1 , Figure 1: is a schematic diagram of the composition of the robot control system provided by the present application, and the robot control system 100 includes a decision-making layer text generation large model 101, an execution layer multimodal large model 102, a ROS main control unit 103 and an execution unit 104. In particular, the possible deployment methods of the robot control system 100 include: the decision-making layer text generation large model 101 and the execution layer multimodal large model 102 can be deployed in the cloud, and the ROS main control unit 103 and the execution unit 104 are deployed on the end-side robot. Or the decision-making layer text generation large model 101, the execution layer multimodal large model 102, the ROS main control unit 103 and the execution unit 104 are all deployed on the end-side robot.

[0038] Among them, the decision-making layer text generation large model 101 is used to parse user instructions and obtain the task sequence corresponding to the user instructions. The execution layer multimodal large model 102 is used to determine the current sub-task based on the task sequence and generate an action list for the current sub-task. The Robot Operating System (ROS) main control unit is the "central nervous system" of the system, responsible for managing the communication and coordination between nodes in the entire ROS network. It is used to call the execution unit for execution based on the action list. The execution unit includes equipment such as the robot's motion chassis, robotic arms, and speakers to perform corresponding actions. In a specific implementation, the robot control system 100 may also include an environmental perception unit, which includes a lidar, an inertial measurement unit (IMU), and a depth camera, etc., to obtain environmental perception information. The environmental perception information includes environmental image information, robot motion state information, etc., and the obtained relevant information is fed back to the execution layer multimodal large model 102 to adjust and optimize the action list.

[0039] See below Figure 2 , Figure 2 1 is a flow chart of a robot task data processing method based on a dual-model architecture provided by the present application. The robot task data processing method based on a dual-model architecture is applied to the robot control system 100, and specifically includes the following steps.

[0040] S210 , the ROS main control unit obtains a text recognition result corresponding to a first user instruction, creates a first context information set for a current task cycle, and adds the text recognition result to the first context information set to obtain a second context information set.

[0041] The first context information set includes a prompt, and the second context information set includes a prompt and a text recognition result.

[0042] In a possible embodiment, the robot control system further includes a speech recognition model, and the ROS main control unit obtains a text recognition result corresponding to the first user instruction, including: after the ROS main control unit detects a preset wake-up word, it obtains user voice information; the speech recognition model determines the first user instruction based on the user voice information uploaded by the ROS main control unit, and generates a text recognition result corresponding to the first user instruction.

[0043] The robot control system also includes a voice module. The system first monitors user voice input through the voice module. Upon detecting a preset wake-up word, the subsequent voice processing process is activated. At this point, the voice module enters the voice activity detection (VAD) phase, filtering out valid voice segments from the user's speech. The system can identify and exclude background noise and irrelevant voice signals, ensuring the accuracy of the voice data. Valid voice segments are saved as WAV-formatted audio files for subsequent processing and analysis. Next, the speech recognition model receives the WAV audio file and parses it, converting it into text. This process typically involves the collaborative work of acoustic models, language models, and speech decoding to accurately identify the text of the user's command. The resulting text recognition results are uploaded to the ROS master control unit, which then performs actions based on the command content, such as controlling robot movement, performing specific tasks, or interacting with other system components.

[0044] It can be seen that in this embodiment, obtaining text recognition results based on the above method enables the control system to respond to the user's voice commands more efficiently, improve user experience, and maintain high recognition accuracy in different environments.

[0045] S220 , the decision-level text generation model outputs a first task sequence according to the text recognition result, and adds the first task sequence to the second context information set to obtain a third context information set.

[0046] Among them, the large model for text generation at the decision-making level can use powerful natural language processing capabilities to understand the user's intentions and convert them into specific actions that the machine can perform. This process usually relies on deep learning models, especially large-scale language models, which have been trained with a large amount of data and can efficiently understand the context, grammar, and logical relationships of common instructions. The generated task sequence will list in detail every operation that the robot needs to perform and include the necessary parameters or conditions to ensure the accurate completion of the task. For example Figure 3As shown, the first user instruction is "Can you help me get a bottle of mineral water from the kitchen?" After receiving the first user instruction, the robot's ROS main control unit obtains the corresponding text recognition result, and then inputs the text recognition result into the decision-making layer large model, that is, the decision-making layer text generation large model, and then obtains the first task sequence output by the decision-making layer text generation large model, including: 1. Record the current location, 2. Navigate to the tea room, 3. Check the location of the mineral water, 4. Pick up the mineral water, 5. Return to the starting position, and 6. Put down the mineral water.

[0047] S230, the execution layer multimodal large model determines the current task from the first task sequence according to the third context information set, generates a first action list corresponding to the current task, and adds the first action list to the third context information set to obtain a fourth context information set.

[0048] Among them, the first action list includes function call information and voice reply information of the action library function. At the execution layer, the system generates the first task sequence output by the large model through the multimodal large model to parse the decision layer text, and determines the current task from the first task sequence based on the third context information set. For example, based on the execution status feedback information of the last task recorded in the third context information set, it is determined whether the last task has been completed. If the execution is completed, the next task of the last task in the task sequence is determined to be the current sub-task. If the execution of the last task fails, it is determined that the current sub-task is still the last task, that is, the task is re-executed.

[0049] This large multimodal model not only understands textual content but also comprehensively determines task priorities and execution strategies based on multiple modalities, including environmental perception, visual input, and tactile feedback. Based on the complexity and specific requirements of the task, the execution layer generates a first action list for the current task. This list contains multiple actions, each corresponding to a specific function or behavior, such as calling a specific action library function to perform a mechanical operation or interacting with the user through a voice response module. The action library function call information clearly instructs the robot on how to operate each hardware module, such as moving, grasping, scanning, or other customized actions. The voice response information ensures that the robot can effectively communicate with the user during execution, providing feedback, asking for confirmation, and expressing progress. Depending on the current task, the first action list can be continuously updated to adapt to environmental changes or user needs, thereby ensuring efficient execution and accurate response in dynamic scenarios.

[0050] S240: The ROS master control unit drives the execution unit to execute the first action list, obtain execution status feedback information, and add the execution status feedback information to the fourth context information set to obtain a fifth context information set.

[0051] The ROS master control unit drives the execution unit to initiate corresponding actions based on the first action list output by the execution layer multimodal large model, and tracks the task execution status in real time. Upon receiving the instructions, the execution unit completes the predetermined operations according to the instructions in the action list. These operations may involve complex tasks such as robot arm movement, motion, visual acquisition, and sensor data processing. During execution, the execution unit continuously provides feedback on execution status to the ROS master control unit. This feedback includes information on whether the current task has been successfully completed, whether any obstacles have been encountered, the system's health status, and environmental perception information acquired during execution. Based on this feedback information uploaded by the ROS master control unit, the execution layer multimodal large model can assess execution progress and make necessary adjustments or optimizations based on actual conditions. If an anomaly is detected during execution, the execution layer multimodal large model can respond promptly through status feedback, such as instructing the re-execution of the action, re-adjusting the action, issuing new instructions, or notifying the user of the progress of the current task. This closed-loop control mechanism ensures that the robot can perform tasks efficiently and accurately, and can respond flexibly to environmental changes or complex situations.

[0052] For example Figure 4 As shown in the figure, the execution layer model (i.e., the execution layer multimodal model) sends an action list to the robot's ROS master control unit. The ROS robot calls the execution unit to execute the actions in the action list and provides feedback on the action execution status and the environment. The action list includes the execution action and the voice response, i.e., "action": ["get_current_pose()"], and "response": "I have memorized my current position."

[0053] See also Figure 5The context information set (i.e., the historical conversation context in the figure) first includes the initial conversation history of the prompt. After receiving the user's instruction, the decision-layer model adds the user instruction and task steps to the conversation history. Specifically, the text recognition results corresponding to the user instruction and the generated task sequence (i.e., the broken-down task steps) are added to the context information set. At this point, the conversation history includes the prompt, user instruction, and broken-down task steps. The execution-layer model then generates the first action list for the current task (the model's response) and adds it to the context information set (i.e., the field model's response is added to the conversation history). After the ROS master unit calls the execution unit to execute the first action list, the resulting execution status feedback (i.e., robot feedback) is also added to the conversation history. After the current task is completed, when the next task begins, the context information set, including the execution status feedback from the previous task, is sent again to the execution-layer multimodal model. In other words, each request to the execution-layer model sends the entire conversation history.

[0054] S250: The execution layer multimodal large model determines whether a preset condition is met according to the execution status feedback information.

[0055] In a possible embodiment, satisfying the preset conditions includes any one of the following: obtaining a target user instruction, wherein the target user instruction is used to instruct the robot to stop working; the execution status feedback information indicates that the robot has failed the current task for a preset number of consecutive times; the execution layer multimodal large model determines that the current task is an unfinishable task.

[0056] If the current task is the first task in the first task sequence, then if the execution layer model does not detect any of the following situations: (1) the user requests to end the current task cycle or a voice command that implies that the robot is no longer needed; (2) the robot fails to execute the action of the current sub-task link multiple times (for example, 2 times); (3) the robot detects that the task of the current sub-task link is an unfinishable task; then, mark the current sub-task link as the second task in the first task sequence; generate the first action list corresponding to the second task, execute the first action list, and record the execution status feedback information. Repeat the above steps until all tasks in the current sub-task cycle are completed.

[0057] The execution-layer multimodal model determines that the current task is unachievable by: First, the execution-layer multimodal model can analyze execution status feedback, particularly the robot's error reports and failure counts. If the robot fails multiple times (e.g., two or more times) while executing the current task and is unable to overcome environmental factors or operational limitations, the execution-layer multimodal model can determine that the task is unachievable. Second, the execution-layer multimodal model can also check whether there are external conditions that affect task completion, such as changes in the robot's required environmental conditions or malfunctioning hardware or sensors that the current task relies on. If any hardware failure or resource unavailability is detected, the execution-layer multimodal model can also determine that the task cannot proceed. Furthermore, the execution-layer multimodal model can also evaluate the logic and feasibility of the task. For example, in certain cases, if the task itself cannot be implemented due to technical limitations or design errors, the main control unit can also make a "task unachievable" decision.

[0058] As can be seen, in this embodiment, determining whether the robot should suspend processing tasks in the current task cycle based on preset conditions allows for timely assessment of obstacles, failures, or incomplete tasks encountered by the robot during execution, allowing for reasonable judgment and adjustments. When the robot is executing the current task and a preset condition is triggered, such as continuous failures in a task link, external interference, or hardware issues, the execution layer's multimodal large model quickly makes a decision on whether to suspend the current task cycle by analyzing this feedback information. Through this mechanism, the system can promptly respond to environmental changes, increased task difficulty, or technical obstacles, preventing the robot from pointlessly continuing to execute unfinished tasks, thereby avoiding resource waste and reduced efficiency, and ensuring that the robot can perform task scheduling in a more efficient and intelligent manner. After pausing the current task cycle, the system can reassess the task, adjust the execution strategy, or redirect attention to other tasks to ensure the successful completion of the overall task. Furthermore, the robot can also automatically resume or switch to an alternative task at the appropriate time based on the system's fault-tolerance mechanism, further improving the flexibility and fault-tolerance of task execution.

[0059] S261 : If not satisfied, use the fifth context information set as the third context information set and repeat S230 - S250 until the current task cycle is completed.

[0060] S262: If the conditions are met, the system enters a waiting state, and when a temporary instruction is obtained in the waiting state, the execution layer multimodal large model determines a second task sequence according to the fifth context information set and the temporary instruction.

[0061] The second task sequence is used to complete the second user instruction. That is, after the execution layer multimodal large model determines the second task sequence, it adds the second task sequence to the fifth context information set to obtain a sixth context information set. Based on the sixth context information set, a second action list for the current task is generated from the second task sequence. The ROS main control unit calls the execution unit to execute the second action list, obtains execution feedback information, and writes the execution feedback information into the sixth context information set to obtain a seventh context information set. The seventh context information set is then used as the fifth context information set. The above steps are repeated until each task in the second task sequence is completed or a preset condition is met, and the current task cycle is determined to be over.

[0062] It can be seen that in this embodiment, the dual-model combined with dynamic reasoning solves the shortcomings of the current single AI large model control method, which is prone to modal interference and difficult training when processing complex tasks.

[0063] In a possible embodiment, before the execution layer multimodal large model determines the second task sequence according to the fifth context information set and the temporary instruction when the preset condition is met, the method also includes: determining whether a second user instruction is obtained in the waiting state; when the second user instruction is obtained, the ROS main control unit determines whether the second user instruction is a temporary instruction; the execution layer multimodal large model determines the second task sequence according to the fifth context information set and the temporary instruction, including: when the second user instruction is a temporary instruction and the preset condition is not met, the execution layer multimodal large model parses the temporary task corresponding to the temporary instruction, and generates a second task sequence for the temporary task according to the temporary task and the fifth context information set.

[0064] Among them, when the execution layer multimodal large model recognizes that all tasks in the first task sequence are marked as completed, it switches to a waiting state and waits for the user's second user instruction. The ROS main control unit determines whether the second user instruction is a temporary instruction. If the second user instruction is a temporary instruction corresponding to a temporary task and the preset conditions are not currently met, the temporary instruction will no longer pass through the decision layer text generation large model, but will be directly passed to the execution layer multimodal large model, and the execution layer multimodal large model will execute the temporary instruction. Whether the second user instruction is a temporary instruction can be determined by a preset flag. If the "new_order_cycle flag" in the program is False, the second user instruction is considered to be a temporary instruction.

[0065] Temporary commands can be relatively simple interactive instructions, such as instructing the robot to "end the current task, rest, or exit," "repeat the action just performed," or "ask about the image you just saw." For example, after waking up the robot, the first user command is: "Look at the objects in front of you, then move forward one meter, move back one meter, and finally perform a dance." After the robot completes the actions in the command and the first task sequence, it is woken up again and issued a temporary command: "Tell me what actions you just performed and what objects you just saw." The command is then directly passed to the execution layer model, which responds to the user based on the information in the context information set.

[0066] It can be seen that in this embodiment, after each task in the current task cycle is completed, it enters a waiting state and determines whether a temporary task is obtained. The temporary task is executed directly through the multimodal large model of the execution layer, without going through the text generation model of the decision layer, which can improve the flexibility and efficiency of task response. In this way, the robot can receive and process new task instructions in real time without waiting for the entire decision-making process, thereby shortening the task processing time. In this mode, the large model of the execution layer will directly parse and execute temporary tasks, relying on multimodal information and environmental feedback to ensure that the robot can quickly adapt to task changes and optimize the operation process. This adjustment enables the robot to operate flexibly in a dynamic environment and can quickly switch to a new task after the original task is completed, thereby improving the robot's adaptability and response speed. In addition, the direct operation of the execution layer can avoid delays in the decision layer, ensuring that the robot can respond to sudden tasks more efficiently and further improve the overall performance of the system.

[0067] In a possible embodiment, the execution layer multimodal large model parses the temporary task corresponding to the temporary instruction, and generates a second task sequence for the temporary task based on the temporary task and the fifth context information set, including: parsing the temporary instruction to obtain the temporary task corresponding to the temporary instruction, the temporary task including a task object and a task requirement; matching the task object in the fifth context information set to obtain a matching result; determining the current state of the task object based on the matching result; and generating the second task sequence for the temporary task based on the current state, the task requirement and the fifth context information set.

[0068] When generating the second task sequence corresponding to the temporary task, the execution layer multimodal large model can match the parsed task object with the context information set to obtain the current state of the task object. Then, the second task sequence is generated based on the current state, the task requirements, and the fifth context information set.

[0069] The second task sequence of the temporary task is generated according to the current state, the task requirements and the fifth context information set, specifically including: first, the execution layer multimodal large model can confirm whether the execution conditions of the temporary task are met by comprehensively evaluating the current state of the task object. This step includes analyzing the availability of the task object, whether the state is in a suitable working state, and whether there are potential problems or obstacles that affect the progress of the task. Based on these evaluation results, it can be determined whether the task object needs to be preprocessed or repaired to ensure that the task can start in the best state.

[0070] The system can then match task requirements with the state of the task object, ensuring consistency between task objectives and available resources. For example, if the task requires object handling or path planning, the model can update the handling strategy based on the current location and state of the task object, avoiding invalid operations or path conflicts during task execution. Furthermore, based on feedback from the fifth context information set, the model can adjust the priority and steps of task execution in real time, quickly adapting to any possible environmental changes or operational errors.

[0071] Finally, the generated second task sequence will formulate an optimal execution plan based on this comprehensive information, including the task objectives, the status of the task objects, feedback from the current environment, and possible obstacles or changing factors. This not only maximizes execution efficiency but also avoids task failures or delays caused by overlooking certain details. Through this flexible and dynamic adjustment mechanism, the execution-layer multimodal large model ensures efficient and sustainable task execution, optimizing the overall task execution process.

[0072] It can be seen that in this embodiment, the second task sequence is generated based on the context information set, so that the execution layer multimodal large model can not only respond to temporary tasks efficiently, but also ensure the high compatibility between tasks and the environment during execution, thereby optimizing task execution efficiency.

[0073] In a possible embodiment, generating the second task sequence of the temporary task based on the current state, the task requirements and the fifth context information set includes: determining the task type of the temporary instruction based on the task requirements; determining whether there is a reference task with the same task type as the temporary instruction in the first user instruction based on the fifth context information set; if so, obtaining the reference task sequence corresponding to the reference instruction in the fifth context information set; generating the second task sequence of the temporary task based on the current state, the task instruction and the reference task sequence.

[0074] It can be determined whether the task included in the first user instruction has a reference instruction with the same task type as the temporary task. If so, the reference task sequence corresponding to the reference instruction in the first task sequence is determined from the fifth context information set, and then a second task sequence is generated based on the reference task sequence. For example, if the task requirement of the temporary task is to pick up or move an item, if the first user instruction includes an item moving or picking task, then the first user instruction is a reference instruction. The reference task sequence can be determined based on the first task sequence, and the steps in the reference task sequence correspond to the item moving or picking task. For example: After waking up the robot, the first user instruction is: "Move the red square from room 1 to room 2." After the task is completed, the robot is woken up again, and the temporary instruction is: "Please move the blue square from room 2 to room 3." At this time, the task type of both the first user instruction and the temporary instruction is an item moving task. The temporary instruction is then directly passed to the execution layer model, and the execution layer multimodal large model can refer to the first task sequence of the first user instruction in the learning context information set to generate the second task sequence.

[0075] The second task sequence of the temporary task is generated based on the current state, the task requirements, and the reference task sequence, specifically including the following steps: comparing and adjusting the step sequence of the reference task based on the current state and task requirements of the temporary task. Specifically, the execution layer multimodal large model can analyze the progress of the task objectives in the current state, such as the current location of the task object, environmental conditions, existing resources, etc., and dynamically adjust the reference task sequence based on this information. If the current state changes (such as changes in environmental conditions, changes in the location of the task object, etc.), the relevant steps in the reference task sequence can be updated to ensure that they adapt to the current actual situation.

[0076] If the current state changes, such as environmental conditions or a shift in the position of the task object, sensors or the fifth context information set can be used to detect and analyze the changes in real time, clarifying the specific content and scope of the change. Subsequently, the current state is compared with the pre-set state in the reference task sequence to identify the affected steps. For these steps, their execution order can be readjusted, or task steps can be replaced based on the latest environmental and task object states. For example, if the original path is blocked, the execution layer's multimodal large model will replan the path and update the relevant action steps. If the task object's position changes, the specific steps for grasping or interacting are updated. Simultaneously, existing resources and conditions can be revalidated to ensure they still meet the task requirements. If not, preparation or waiting steps can be added to ensure the effective execution of subsequent actions. Furthermore, action execution parameters such as force, speed, or angle can be adjusted based on the new state to improve the success rate of the action. Throughout this process, execution performance is continuously monitored based on the context set, and steps are continuously optimized based on feedback, forming a closed-loop dynamic adjustment mechanism. This ensures that the task sequence can flexibly adapt to environmental changes and ensure the successful completion of the task.

[0077] As can be seen, in this embodiment, directly referencing the first task sequence to generate a second task sequence for temporary actions not only significantly improves the efficiency of task execution, but also ensures that the task maintains consistency and stability during continuous execution. By borrowing the execution logic and operation steps from the first task sequence, it is possible to quickly adapt to new task requirements, avoid replanning each task step, and thus reduce delays in task execution. In this way, the execution of temporary tasks is more efficient and can be optimized by leveraging the successful experience and feedback from previous task execution.

[0078] Furthermore, generating a second action list based on the reference task sequence ensures contextual consistency during task execution, avoiding unnecessary complexity or errors caused by redesigning task steps. The execution layer can automatically adjust the action sequence and execution strategy based on feedback from the first action list, ensuring that the new task can be carried out smoothly within the existing environment and processes. This approach also reduces manual intervention during task execution, improves the system's autonomy and intelligence, and further enhances the robot's adaptability and execution capabilities in dynamic environments.

[0079] In a possible embodiment, the second task sequence of the temporary task is generated according to the current state, the task requirements and the reference task sequence, including: determining the initial position of the task object according to the current state; determining the target position of the task object according to the task requirements; obtaining a map mapping table of the site where the robot is located; determining the initial mapping area of ​​the initial position of the task object and the target mapping area of ​​the target position according to the map mapping table; obtaining the reference initial mapping area and reference target mapping area of ​​the task object in the reference instruction; replacing the reference initial mapping area of ​​the task object in the reference task sequence with the initial mapping area, and replacing the reference target mapping area with the target mapping area, to obtain a second task sequence.

[0080] Among them, such as Figure 6 As shown, when the robot is navigating, it can establish a mapping relationship between the grid map and the real environment area to obtain a map mapping table. This mapping table can divide different places in the map into different areas (for example, Figure 6 Then, the robot maps the real-world areas to the alphabetical symbols (areas 1-5 in the model). The YAML file then writes the map coordinates for the alphabetical symbols. When the robot is instructed to navigate to the actual area, it simply converts the large model into the corresponding alphabetical symbols, allowing it to understand the real-world location.

[0081] In the above example, the temporary instruction is: "Please take the blue block in room 2 to room 4". At this time, the execution layer multimodal large model first parses the "initial mapping area and target mapping area" of the temporary instruction as "area 1 and area 2", "reference initial mapping area and reference target mapping area" as "area 2 and area 4", the task requirement is "pick up items", and the "task object" is "blue block"; then, according to "area 1 and area 2", the "map mapping table" is queried to determine whether the location is reachable; if it is determined that the location is unreachable, the unfinishable task processing branch is executed, and it is considered that the preset conditions are met; and, it is determined that "pick up items" is a complete action; and, the fifth upper and lower parts are queried. The environmental object identification result in the execution status feedback information of the reference task in the fifth context information set is used to determine that the "blue square" is the environmental object existing in the "area 2" that has been identified and stored in this task cycle; then, the temporary instruction is compared with the task type of the reference task in the reference instruction in the fifth context information set, and the first task sequence of the reference task in the fifth context information set is called, and the original "area 1" in the "first task sequence" is updated to "area 2", and the original "area 2" is updated to "area 4", and the updated "second task sequence" is executed until the task execution is completed.

[0082] For example Figure 7As shown, the robot moves based on the second action list corresponding to the second task sequence. Figure 7 The robot is shown in Figure 6 The process of moving from area 2 to area 4 after getting the blue block is shown in the figure. Figure 7 It can also be found that there are obstacles and other contents in the real environment. At this time, the ROS main control unit can also obtain the visual and depth information obtained by the depth camera, and determine the moving path of the mobile phase area 4 based on the visual and depth information combined with the grid map, and then call the motion chassis in the execution unit to move according to the moving path.

[0083] It can be seen that in this embodiment, by replacing the mapping area in the first action list based on the task requirements and the map mapping table, the second action list can be quickly generated and the executability of the actions can be ensured.

[0084] In a possible embodiment, the method further includes: when no second user instruction is obtained in the waiting state, or when the second user instruction is not a temporary instruction, or when the temporary instruction indicates that the preset condition is met, calling a completion dialogue function to reset the fifth context information set and determining that the current task cycle ends.

[0085] Among them, when the preset conditions are met, if the second user instruction is not obtained, the fifth context information set is reset. Or when the preset conditions are met, if the second user instruction is not a temporary instruction, the fifth context information set is also reset. In particular, if the second user instruction indicates that the preset conditions are met, the fifth context information set is directly reset and the current task cycle is determined to be over. For example Figure 8 As shown, if any of the following conditions are met, the execution layer model calls the finish_dialogue function, resets the conversation history context (sets the fifth context information set), ensures that the initial conversation history in the context of the conversation history includes the prompt, and starts a new task cycle. Case 1: The user requests to end the current task cycle; Case 2: The robot fails to contact the user twice while performing an action; Case 3: The robot encounters an unfinishable task. For example, after waking up the robot, the first user instruction is "Take the red block from room 1." The task is completed. When the robot is woken up again, the temporary instruction is "Take a break" or "End the current task." In this case, the instruction is directly passed to the execution layer multi-module model, which calls the finish_dialogue() function in the generated action list. The finish_dialogue() function resets the context information set and starts a new task cycle.

[0086] As can be seen, in this embodiment, if the preset conditions are met, the context information set is reset and a new task cycle is initiated. This ensures that the system has a clear initial state when facing a new task, preventing residual effects of previous tasks from interfering with the execution of the new task. By resetting the context information set, the robot can handle the new task from scratch, reassess the current environment and task requirements, and optimize the execution process and resource allocation. This process improves the system's adaptability to task changes and avoids misjudgments or ineffective execution caused by historical data or outdated task states.

[0087] See also Figure 9 and Figure 10 , taking the decision-making layer and execution layer as the cloud-based AI model and the ROS master control unit located in the end-side robot as an example, the overall process of this application is introduced in detail.

[0088] like Figure 9 As shown, when a user command includes a preset wake-up word, the voice module of the on-premises robot is awakened based on the keyword. Then, through VAD (Vertical Audio Device) detection and dynamic recording, the first user command is sent to the ROS master control unit. The ROS master then uploads the audio file to the speech recognition model in the cloud-based AI model, generating a text recognition result. The ROS master then uploads the user command to the decision-layer text generation model, generating a task sequence. The execution layer multimodal model then generates an action list based on the task sequence, and the ROS master calls the execution unit to execute the action list. When voice output is required, the text corresponding to the execution result is uploaded to the speech synthesis model, which generates audio and outputs it through the voice module's speakers. The on-premises robot's execution unit includes a motion chassis and a robotic arm. The ROS master control unit can obtain the hardware status of the motion chassis and robotic arm from the STM32 control board and send control parameters to control hardware execution. At the same time, the end-side robot also includes a depth camera, lidar, and inertial measurement unit (IMU) to obtain visual, depth information, and robot motion status information, and generate execution status feedback information based on the task execution results. When feeding back the execution results, feedback status vision can be included. The ROS main control unit uploads the execution status feedback information to the execution layer multimodal large model for action list generation.

[0089] like Figure 10As shown, within a single task cycle, the decision-layer text generation model calls the robot action function library training examples from the RAG database based on the text recognition text corresponding to the user command. It understands the user command, plans the task, and generates task steps (i.e., a task step list), which includes several subtasks. The task steps are then input into the execution layer multimodal model. The execution layer multimodal model calls the robot action function library and map mapping based on the prompt word, generates an action list and response content, and then the robot executes the action based on the action list and response content, generating action execution status feedback and visual information that are fed back to the multimodal model to execute subsequent tasks.

[0090] The present application also provides a robot control system, including a decision-making layer text generation large model, an execution layer multimodal large model, a ROS main control unit and an execution unit; the decision-making layer text generation large model is used to execute the steps executed by the decision-making layer text generation large model in any of the above embodiments; the execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any of the above embodiments; the ROS main control unit is used to execute the steps executed by the ROS main control unit in any of the above embodiments; the execution unit is used to execute the steps executed by the execution unit in any of the above embodiments.

[0091] The present application also provides a robot, including a decision-making layer text generation large model, an execution layer multimodal large model, a ROS main control unit and an execution unit; the decision-making layer text generation large model is used to execute the steps executed by the decision-making layer text generation large model in any of the above embodiments; the execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any of the above embodiments; the ROS main control unit is used to execute the steps executed by the ROS main control unit in any of the above embodiments; the execution unit is used to execute the steps executed by the execution unit in any of the above embodiments.

[0092] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of this application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0093] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0094] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0095] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0097] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0099] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program code.

[0100] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing related hardware. The program can be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0101] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A robot task data processing method based on a dual-model architecture, characterized in that: The decision-making layer text generation large model, execution layer multimodal large model, ROS master control unit and execution unit used in the robot control system include: Step a: The ROS main control unit obtains a text recognition result corresponding to a first user instruction, creates a first context information set for a current task cycle, and adds the text recognition result to the first context information set to obtain a second context information set; Step b: the decision-level text generation model outputs a first task sequence according to the text recognition result, and adds the first task sequence to the second context information set to obtain a third context information set; Step c: the execution layer multimodal large model determines a current sub-task from the first task sequence based on the third context information set, generates a first action list corresponding to the current sub-task, and adds the first action list to the third context information set to obtain a fourth context information set; Step d: the ROS master control unit drives the execution unit to execute the first action list, obtains execution status feedback information, and adds the execution status feedback information to the fourth context information set to obtain a fifth context information set; Step e: the execution layer multimodal large model determines whether a preset condition is met based on the execution status feedback information; If not, the fifth context information set is used as the third context information set, and steps c to e are repeated until the current task cycle is completed; If the conditions are met, the system enters a waiting state, and when a temporary instruction is obtained in the waiting state, the execution layer multimodal large model determines a second task sequence according to the fifth context information set and the temporary instruction.

2. The method according to claim 1, characterized in that Before the execution layer multimodal large model determines the second task sequence according to the fifth context information set and the temporary instruction, the method further includes: determining whether a second user instruction is obtained in the waiting state; In the case of acquiring the second user instruction, the ROS main control unit determines whether the second user instruction is a temporary instruction; The execution layer multimodal large model determines a second task sequence according to the fifth context information set and the temporary instruction, including: When the second user instruction is a temporary instruction and does not meet the preset condition, the execution layer multimodal large model parses the temporary task corresponding to the temporary instruction and generates a second task sequence for the temporary task based on the temporary task and the fifth context information set.

3. The method according to claim 2, characterized in that The execution layer multimodal large model parses the temporary task corresponding to the temporary instruction, and generates a second task sequence for the temporary task according to the temporary task and the fifth context information set, including: Parsing the temporary instruction to obtain a temporary task corresponding to the temporary instruction, wherein the temporary task includes a task object and a task requirement; matching the task object in the fifth context information set to obtain a matching result; Determine the current state of the task object according to the matching result; A second task sequence for the temporary task is generated according to the current state, the task requirement, and the fifth context information set.

4. The method according to claim 3, characterized in that Generating the second task sequence of the temporary task according to the current state, the task requirement, and the fifth context information set includes: Determining a task type of the temporary instruction according to the task requirement; determining, according to the fifth context information set, whether there is a reference instruction in the first user instruction having the same task type as the temporary instruction; If so, obtaining a reference task sequence corresponding to the reference instruction in the fifth context information set; A second task sequence for the temporary task is generated according to the current state, the task requirement, and the reference task sequence.

5. The method according to claim 4, characterized in that Generating the second task sequence of the temporary task according to the current state, the task requirement, and the reference task sequence includes: determining an initial position of the task object according to the current state; Determine the target location of the task object according to the task requirements; Get the map of the robot's location. Determine an initial mapping area of ​​an initial position of the task object and a target mapping area of ​​a target position according to the map mapping table; Acquire a reference initial mapping area and a reference target mapping area of ​​the task object in the reference instruction; The reference initial mapping area of ​​the task object in the reference task sequence is replaced by the initial mapping area, and the reference target mapping area is replaced by the target mapping area, to obtain a second task sequence.

6. The method according to claim 2, characterized in that The method further comprises: When the second user instruction is not obtained in the waiting state, or when the second user instruction is not a temporary instruction, or when the temporary instruction indicates that the preset condition is met, the completion dialogue function is called to reset the fifth context information set and determine that the current task cycle is ended.

7. The method according to claim 1, characterized in that Meeting the preset conditions includes any of the following: Obtaining a target user instruction, where the target user instruction is used to instruct the robot to stop working; The execution status feedback information indicates that the robot has failed the current task for a preset number of consecutive times; The execution layer multimodal large model determines that the current task is an unfinishable task.

8. The method according to claim 1, characterized in that The robot control system further includes a speech recognition model, and the ROS main control unit obtains a text recognition result corresponding to the first user instruction, including: After the ROS main control unit detects the preset wake-up word, it obtains the user's voice information; The speech recognition model determines the first user instruction based on the user voice information uploaded by the ROS main control unit, and generates a text recognition result corresponding to the first user instruction.

9. A robot control system, characterized in that: It includes a large text generation model at the decision layer, a large multimodal model at the execution layer, a ROS master control unit, and an execution unit; The decision-making layer text generation large model is used to perform the steps performed by the decision-making layer text generation large model in any method of claims 1-8; The execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any one of the methods of claims 1 to 8; The ROS master control unit is configured to execute the steps executed by the ROS master control unit in any one of the methods of claims 1 to 8; The execution unit is used to execute the steps executed by the execution unit in any one of the methods according to claims 1-8.

10. A robot, characterized in that: It includes a large text generation model at the decision layer, a large multimodal model at the execution layer, a ROS master control unit, and an execution unit; The decision-making layer text generation large model is used to perform the steps performed by the decision-making layer text generation large model in any method of claims 1-8; The execution layer multimodal large model is used to execute the steps executed by the execution layer multimodal large model in any one of the methods of claims 1 to 8; The ROS master control unit is configured to execute the steps executed by the ROS master control unit in any one of the methods of claims 1 to 8; The execution unit is used to execute the steps executed by the execution unit in any one of the methods according to claims 1-8.

Images