Humanoid robot-oriented multi-agent cooperation method and system

Abstract

The invention provides a multi-agent cooperation method and system for a humanoid robot, and belongs to the field of multi-agent collaboration.The method comprises the steps that in response to a received target task described by a natural language, the target task is analyzed, and a subtask sequence is obtained; determining a structured sub-task sequence and a robot role allocation strategy based on the type of each sub-task in the sub-task sequence; packaging the structured sub-task sequence and the robot role allocation strategy to generate a task state; the task state and the environment feedback data are aligned, and a joint vector is constructed; retrieving at least one motion segment matched with the joint vector from a predefined reference motion data set, and generating a robot cooperation instruction based on the at least one motion segment; and controlling at least one execution robot to execute corresponding motion according to the robot cooperation instruction. According to the humanoid robot-oriented multi-agent cooperation method and system provided by the invention, the generalization ability of multi-robot cooperation can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-agent collaboration, and more particularly to a multi-agent collaboration method and system for humanoid robots. BACKGROUND

[0002] With the continuous development of intelligence, more and more scenarios can be executed by robots. When deploying humanoid agents to perform multi-task operations in complex environments, the intelligent system needs to have both natural and smooth basic motion capabilities and understanding capabilities for long sequence task structures.

[0003] The existing paradigm is to use "single task reinforcement learning + artificially designed reward function", that is, for each subtask, an independent policy network is designed; and a reward function is constructed item by item by hand to guide the humanoid agent to complete the action.

[0004] However, the existing method is applicable to a single task or a fixed scene, and lacks the ability to uniformly model and adaptively collaborate in a complex multi-task, multi-role environment. SUMMARY

[0005] The purpose of the present application is to provide a multi-agent collaboration method and system for humanoid robots, which can enable an intelligent system to learn multiple motion skills through the same control framework and improve the generalization ability of the intelligent system in a multi-task scenario.

[0006] In a first aspect, the present application provides a multi-agent collaboration method for humanoid robots, comprising: In response to receiving a natural language description of a target task, the target task is parsed to obtain a subtask sequence corresponding to the target task; Based on the types of each subtask in the subtask sequence, a structured subtask sequence and a robot role allocation strategy are determined, the structured subtask sequence includes a collaboration time sequence of at least one execution robot, and the robot role allocation strategy includes at least one execution robot allocated for each subtask, the at least one execution robot being a humanoid robot with an embedded agent; The structured subtask sequence and the robot role allocation strategy are packaged to generate a task state; Align the task state and the environment feedback data to construct a joint vector; Retrieving at least one motion segment matching the joint vector from a predefined reference motion data set, and generating a robot collaboration instruction based on the at least one motion segment, the predefined reference motion data set including a plurality of motion segments for controlling the execution of robot actions; According to the robot collaboration instruction, at least one execution robot is controlled to perform the corresponding motion.

[0007] A second aspect of this application provides a multi-agent collaborative system for humanoid robots, including: a task decomposition module, a collaborative decision-making module, and a motion control module; The task allocation module is used for: In response to receiving a target task described in natural language, the target task is parsed to obtain a sequence of subtasks corresponding to the target task; The structured subtask sequence and robot role allocation strategy are determined based on the type of each subtask in the subtask sequence. The structured subtask sequence includes the collaborative timing of at least one execution robot. The robot role allocation strategy includes at least one execution robot assigned to each subtask. At least one execution robot is a humanoid robot with a built-in intelligent agent. The structured subtask sequence and robot role assignment strategy are packaged to generate task state; The collaborative decision-making module is used for: Align task status and environmental feedback data to construct a joint vector; Retrieve at least one motion segment that matches the joint vector from a predefined reference motion dataset, and generate robot cooperative instructions based on at least one motion segment. The predefined reference motion dataset includes multiple motion segments that control the execution of robot actions. Motion control module, used for: Control at least one robot to perform the corresponding motion according to the robot coordination instructions.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the multi-agent cooperation method for humanoid robots described above.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-agent cooperation method for humanoid robots described above.

[0010] The beneficial effects of the multi-agent cooperation method and system for humanoid robots provided in this application are as follows: The embodiment of the application obtains a subtask sequence by analyzing a natural language target task, determines a structured sequence and a robot role allocation strategy based on a subtask type, aligns a joint vector constructed by a task state packed and environment feedback data, retrieves a matched motion segment from a predefined motion data set to generate a cooperative instruction, and controls a robot to execute. The problems of poor single task adaptability, lack of multi-role cooperation, and dependence on artificial reward functions in the prior art are solved, multi-motion skill learning under a unified control framework is realized, and the generalization ability and dynamic environment adaptability of a complex multi-task scene are improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A flowchart of a multi-agent cooperation method for a humanoid robot provided by an embodiment of the present application is shown in the figure. Figure 2 A flowchart of a multi-agent cooperation method for a humanoid robot provided by another embodiment of the present application is shown in the figure. Figure 3 A work flowchart of a task decomposition module provided by an embodiment of the present application is shown in the figure. Figure 4 A work flowchart of a cooperative decision module provided by an embodiment of the present application is shown in the figure. Figure 5 A work flowchart of a motion control module provided by an embodiment of the present application is shown in the figure. Figure 6 A structural block diagram of a multi-agent cooperation system for a humanoid robot provided by an embodiment of the present application is shown in the figure. Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0014] In a complex environment multi-task scenario, a humanoid agent needs to have both natural motion control and long sequence task understanding capabilities. The current mainstream solution adopts the "single task reinforcement learning + artificial design reward function" paradigm, which constructs an independent policy network for each sub-task and relies on a hand-crafted reward function to guide action execution. However, this paradigm has significant limitations. Its training goal is limited to a specific task or a fixed scene, and it lacks unified modeling and adaptive collaboration capabilities in complex multi-task and multi-role environments.

[0015] The prior art mainly has the following defects: (1) Reward function design complexity leads to cross-task migration difficulty: the prior art needs to design a reward function for different tasks separately, which highly depends on human experience and has high specialization, making it difficult to reuse the strategy across tasks. When expanding new tasks, the reward structure must be reconfigured and the network must be retrained, significantly increasing the cost of human labor; in a multi-skill scenario, the reward items of different tasks often conflict, making it difficult for strategy training to converge stably, and different motion styles and difficulty levels need to be adjusted independently, making reward engineering a performance bottleneck of the system.

[0016] (2) Multi-agent collaboration dynamic adaptability is insufficient: in a multi-agent collaboration scenario, existing methods design independent reward functions for each agent, which cannot effectively handle task timing dependencies and role specialization requirements. Changes in environmental parameters (such as task order, target position, and number of agents) can easily cause the reward mechanism to fail. The fine-tuned strategy often overfits to a specific scene, with weak generalization ability, making it difficult to support adaptive role switching and collaborative strategy optimization during task execution, and lacking the ability to handle dynamic task structures and long sequence collaboration goals.

[0017] To solve the above problems, the embodiment of the present application provides a multi-agent collaboration method for humanoid robots, which enables agents to learn multiple motion skills through a unified control framework without the need to construct independent reward functions for each task or train multiple policy networks separately, and enables multi-role intelligent collaboration in complex environments, significantly improving the generalization ability, execution efficiency and training stability of the system in a multi-task scenario.

[0018] To make the purpose, technical solutions and advantages of the present application clearer, specific embodiments will be described below with reference to the drawings.

[0019] The multi-agent collaboration method for humanoid robots provided by the embodiment of the present application can be applied to an intelligent system, which includes a task decomposition module, a collaborative decision-making module and a motion control module.

[0020] Please refer to Figure 1 , Figure 1A flowchart of a multi-agent collaboration method for a humanoid robot provided by an embodiment of the present application can be executed by an intelligent system, and the method can include S101-S106.

[0021] In embodiments of the present application, S101-S103 can be executed by a task decomposition module in the intelligent system. S104-S105 can be executed by a collaborative decision module in the intelligent system. S106 can be executed by a motion control module in the intelligent system. The task decomposition module can also be considered a large model task decomposition module with large language model capabilities, and the motion control module can also be considered a general motion controller that controls the actions of the robot.

[0022] The execution process of the method of the embodiments of the present application is described in detail below.

[0023] S101, in response to receiving a natural language description of a target task, parsing the target task to obtain a subtask sequence corresponding to the target task.

[0024] In embodiments of the present application, the intelligent system can interface a large language model (LLM) through an API interface, and the task decomposition module can specifically interface the LLM.

[0025] When the embodiments of the present application receive a natural language description of a carrying task or other task issued by a user or an external control device, the received task can be marked as a target task. After receiving the target task, the embodiments of the present application can call an external large language model through an API interface to parse the target task to obtain a corresponding subtask sequence.

[0026] In embodiments of the present application, the target task can be a text description including multiple actions or requirements, and therefore, semantic parsing of the target task can obtain multiple subtask sequences.

[0027] In embodiments of the present application, the parsing process can be as follows: The target task is semantically parsed based on a predetermined structured prompt to obtain a subtask list corresponding to the target task; the structured prompt is constructed based on a task context, a predefined action primitive library, and an output format constraint.

[0028] The subtask list is parsed to extract a subtask sequence corresponding to the target task.

[0029] An exemplary detailed execution process is as follows: T01, receiving a natural language description of a target task.

[0030] In response to receiving a task described in natural language (e.g., "move a long box from room A to room B through a narrow door gap") issued by a user or an external control device, the task is marked as a target task.

[0031] T02, a large language model (LLM) is called by a task decomposition module.

[0032] The LLM is called by the task decomposition module to analyze the target task.

[0033] T03, semantic analysis based on structured prompts to obtain a subtask list.

[0034] The task decomposition module uses a predetermined structured prompt to guide the LLM to perform semantic analysis on the target task to obtain a subtask list corresponding to the target task (containing multiple action or requirement decomposition items). The construction conditions of the structured prompt include at least one of the following: task context, pre-defined action primitive library (such as "open door", "grab", "cooperative transport", "pose adjustment", etc.), output format constraint, etc.

[0035] T04, parse the subtask list and extract a subtask sequence.

[0036] The subtask list returned by the LLM is format-verified and structurally parsed to finally extract an ordered subtask sequence corresponding to the target task (for example: ① open door → ② transport box through door gap → ③ place at target location).

[0037] The embodiments of the present application use the LLM outside the task decomposition module and utilize structured prompts (combined with task context, pre-defined action primitive library, and output format constraint) to improve the accuracy and flexibility of semantic analysis with the help of large models, and to ensure the standardization of the subtask list through standardized prompts; further extract an ordered subtask sequence from the list to make the task execution logic clear and adapt to the step-by-step requirements of the intelligent system; at the same time, support natural language input from users or external devices, broaden application scenarios, enhance system practicality and interaction friendliness, and lay a reliable foundation for efficient execution of subsequent tasks.

[0038] S102, determine a structured subtask sequence and a robot role allocation strategy based on the type of each subtask in the subtask sequence, the structured subtask sequence including a cooperation timing sequence of at least one execution robot, and the robot role allocation strategy including at least one execution robot allocated for each subtask, the at least one execution robot being a humanoid robot with an embedded agent.

[0039] In the embodiments of the present application, the subtask sequence includes multiple subtasks, and the type of the subtask can include whether two-arm operation is needed or whether narrow space passage is involved, etc.

[0040] The embodiment of the present application can assign at least one execution robot to each subtask according to the type of each subtask, the number of required degrees of freedom, and the available robot resource pool through a rule engine or a lightweight scheduling algorithm to determine a robot role allocation strategy. In addition, the cooperation timing of each execution robot can also be determined to obtain a structured subtask sequence.

[0041] The robot resource pool includes the current position, load capacity, end effector type, and other state information of each execution robot.

[0042] The lightweight scheduling algorithm can include a priority-based task-execution robot matching table.

[0043] For example, the cooperation timing can include: Execution robot 1 is responsible for opening the door, execution robots 2 and 3 cooperatively carry the long box to maintain balance, and execution robot 4 fine-tunes the box pose in real time during the door gap phase to prevent collision.

[0044] S103, packaging the structured subtask sequence and the robot role allocation strategy to generate a task state.

[0045] After obtaining the structured subtask sequence and the robot role allocation strategy, the embodiment of the present application can package the generated structured subtask sequence and the robot role allocation strategy into a standard data structure, such as a tex and JSON object, which is used to indicate the task state of each execution robot.

[0046] In addition, after obtaining the task state, the task decomposition module can send the task state to the cooperative decision module so that the cooperative decision module generates a robot cooperation instruction.

[0047] S104, aligning the task state and the environment feedback data to construct a joint vector.

[0048] The embodiment of the present application can sequentially timestamp align and coordinate-uniformly convert the task state and the environment feedback data to obtain a joint vector.

[0049] In the embodiment of the present application, the task state is generally issued by the task decomposition module and generally includes a current to-be-executed subtask (for example, carrying a box through a door gap), a robot role of an allocated robot (for example, “robot 2 is a main pusher and robot 3 is an auxiliary support”), a task progress stage (for example, “the door has been opened, and the carrying phase is ready”), and a task constraint condition (for example, “the box inclination angle needs to be kept less than 15°”).

[0050] The environment feedback data can be obtained by multiple external perception subsystems.

[0051] Optionally, the robot body state (e.g., joint angles, root pose) is reported by the robot's built-in Inertial Measurement Unit (IMU) and encoders. The box pose and position are estimated by the vision system (e.g., RGB-D camera or visual-inertial odometry) combined with object recognition algorithms. The environment obstacle distribution and door gap geometry are provided by the global map (e.g., point cloud map from Simultaneous Localization and Mapping) and local obstacle avoidance sensors (e.g., LiDAR). The collision risk is calculated by the collision detection module based on the current robot-object-environment geometry model in real time.

[0052] The embodiments of the present application can sequentially timestamp align and coordinate-uniform convert the task state and environment feedback data to jointly constitute a joint vector of the collaborative decision module.

[0053] S105, retrieve at least one motion segment matching the joint vector from the pre-defined reference motion data set, and generate a robot collaborative instruction based on the at least one motion segment, the pre-defined reference motion data set including a plurality of motion segments for controlling the execution of robot actions.

[0054] The matching process of the embodiments of the present application can include: Jointly encode the joint vector and each motion segment in the pre-defined reference motion data set. Obtain a plurality of matching scores according to the adaptation scores of the joint vector and each motion segment obtained according to the joint encoding. Determine at least one motion segment matching the joint vector according to the matching scores.

[0055] The embodiments of the present application can retrieve and match the motion segment of the most suitable task state from the pre-defined reference motion data set. The pre-defined reference motion data set can be constructed by Motion Capture combined with Retargeting technology.

[0056] Construction process: first, collect high-fidelity motion sequences of humans performing typical carrying tasks (e.g., pushing a box, passing through a narrow door sideways), and then map the human motion to the humanoid robot skeleton through a dynamics-consistent retargeting algorithm to generate robot feasible motion data.

[0057] In the embodiments of the present application, each motion segment in the pre-defined reference motion data set contains the following time series data (with a step size of 10ms): Joint Angles (Joint Angles); Joint Angular Velocities (Joint Angular Velocities); Quaternion and linear velocity of the root node (Root Link); Local quaternion and position of body key parts (e.g. hands, feet, torso) relative to parent node.

[0058] wherein each motion segment is further attached with semantic labels (e.g. “side-carry”, “two-person cooperative lifting”, “obstacle avoidance”) and applicable conditions (e.g. door width range, object size interval).

[0059] For example, the matching process of the embodiment of the application is driven by a Multi-Agent Reinforcement Learning (MARL) policy model: the model jointly encodes the joint vector and the semantic labels and initial state features of each candidate motion segment, outputs the adaptation scores (or Q values) of each motion segment, and selects the motion segment with the highest score as the current policy basis.

[0060] For example, upon receiving the sub-task “pass through narrow door gap” and perceiving that the door width < 60 cm, the model preferentially activates the “side-carry” segment.

[0061] Subsequently, based on the selected motion segment, the MARL model further generates specific robot coordination instructions (i.e. “optimized instructions”), including role refinement, action initiation timing, spatial coordination constraints, and synchronization requirements for each robot (for example: “robot 1 immediately retreats to a safe zone after completing the door opening action, robot 2 advances the box from the left side at a 30° inclination, robot 3 provides lateral support on the right side, and both actions need to be synchronized within ± 50 ms”).

[0062] The embodiment of the application can encapsulate the robot coordination instructions as structured messages to send to the motion control module to drive each execution robot to perform the underlying actions.

[0063] In addition, the coordination decision module in the embodiment of the application can continuously receive execution feedback from the general motion controller, including action execution delay, end-of-arm force sensor readings, box pose deviation, and abnormal event flags (such as “box stuck” and “slip overrun”). When the feedback indicates that the current policy is invalid (for example, the box is stalled at the door gap for more than a preset time threshold), the MARL model triggers policy re-planning based on the negative reward signal: re-evaluates the current joint state (including the latest task stage and environmental feedback), switches to an alternative motion segment (such as from “straight-line advance” to “adjust posture + small back-pull”), and updates the subsequent coordination instructions, thereby achieving online dynamic correction of the decision and improving the robustness and adaptability of multi-robot collaboration.

[0064] S106, controlling at least one execution robot to perform the corresponding motion according to the robot coordination instructions.

[0065] The motion control module can control each execution robot to perform the motion corresponding to the motion segment according to the robot coordination instruction after receiving the robot coordination instruction.

[0066] From the above, it can be concluded that the embodiments of the present application obtain a subtask sequence by analyzing a natural language target task, determine a structured sequence and a robot role allocation strategy based on a subtask type, align a joint vector constructed by packaging a task state and environment feedback data, retrieve a matched motion segment from a predefined motion data set to generate a coordination instruction, and control a robot to perform. The problems of poor single task adaptability, lack of multi-role coordination, and dependence on artificial reward functions in the prior art are solved, multi-motion skill learning under a unified control framework is realized, and the generalization ability and dynamic environment adaptability of complex multi-task scenarios are improved.

[0067] In some embodiments of the present application, controlling at least one execution robot to perform corresponding motion according to the robot coordination instruction comprises: performing semantic analysis on the robot coordination instruction to obtain a task description vector.

[0068] determining the most matching action template of the at least one execution robot based on the task description vector.

[0069] controlling the at least one execution robot to perform corresponding motion based on the most matching action template of the at least one execution robot.

[0070] In some embodiments of the present application, controlling the at least one execution robot to perform corresponding motion based on the most matching action template of the at least one execution robot comprises: for each execution robot, generating a discrete action sequence based on the optimal action template of the execution robot, and controlling the execution robot to perform corresponding motion based on the discrete action sequence.

[0071] In embodiments of the present application, the execution robot can serve as a bottom-layer execution unit of a humanoid robot, can receive a robot coordination instruction and generate a motion action sequence conforming to physical laws in real time, and realize plug-and-play calling of carrying skills.

[0072] The execution process of the embodiments of the present application is as follows: A01, performing semantic analysis on the robot coordination instruction.

[0073] Specifically, the semantic analysis includes natural language processing (NLP) on the instruction text, extracting the subject (such as robot identification), action verb (such as “push”), and operation object (such as “box”) therein, and constructing a structured task description vector in combination with context environment information (such as task scene, obstacle layout).

[0074] A02, the task description vector is matched with the metadata tags of each action segment in the predefined reference motion dataset, including action type, applicable object category, typical trajectory characteristics, etc. The most matched action template (such as hand end trajectory, center of gravity movement path, support foot switching timing, etc.) is determined through a pre-trained semantic embedding model or a rule-based mapping table.

[0075] A03, based on the dynamics model customized for the humanoid robot, a real-time action sequence that meets the physical constraints is generated. The dynamics model uses a multi-rigid-body system modeling method, and is expressed in the form of a state space equation as follows:

[0076] wherein, is the joint angle vector, is the inertia matrix, is the Coriolis and centrifugal force term, is the gravity term, and τ is the joint torque input. Although this model is based on classical dynamics theory, it is parameterized and simplified for the humanoid structure (such as biped, 7-DOF arm) of the robot, and is suitable for real-time calculation.

[0077] Based on the above model, the embodiments of the present application can use the action template as the initial reference to generate a time-discretized action sequence through real-time optimization algorithms (such as model predictive control) under the premise of meeting the joint torque limit, ground contact friction cone constraint, zero moment point stability and other physical conditions. The action sequence is composed of a series of time-stamped joint target states, each time step containing the target angle, angular velocity and angular acceleration of each joint (for example, output at a period of 10ms).

[0078] A04, the motion control module can send the action sequence to the robot bottom layer execution unit (including joint servo driver and end effector controller) through a real-time communication bus (such as EtherCAT or CAN), to drive the joint motors and gripper to cooperatively complete the specified moving action (such as smoothly pushing a box through a narrow door gap).

[0079] Exemplary, Figure 2 The flowchart of the multi-agent collaboration method for humanoid robots provided by another embodiment of the present application is shown in FIG. 2, and the execution process of the embodiment of the present application can be as follows: Figure 2 S1: receive a complex task description.

[0080] S2: use a large language model to decompose the task to obtain three sub-tasks. Among them, sub-task 1 is to open the door, sub-task 2 is to carry, and sub-task 3 is to place.

[0081] ​S3: Role assignment. Assign subtask 1 to robot 1, assign subtask 2 to robot 2, and assign subtask 3 to robot 3.

[0082] S4: The collaborative decision module can perform reference motion segment selection, and then control the humanoid robot to perform corresponding actions through the motion controller.

[0083] Exemplary, Figure 3 The workflow diagram of the task decomposition module provided by an embodiment of the present application; Figure 4 The workflow diagram of the collaborative decision module provided by an embodiment of the present application; Figure 5 The workflow diagram of the motion control module provided by an embodiment of the present application.

[0084] As Figures 3 to 5 shown, the division of labor of the large model task decomposition module, the collaborative decision module, and the general motion controller module in the embodiment of the present application is as follows: 1. Working process of the large model task decomposition module: Receive natural language task description.

[0085] Analyze key elements according to spatial obstacles, object characteristics, and environmental constraints.

[0086] Generate a structured subtask sequence according to the decomposition result.

[0087] Dynamically assign robot roles and execution order.

[0088] Output the subtask sequence + role assignment scheme to the collaborative decision module.

[0089] 2. Working process of the collaborative decision module: Receive real-time task status and environmental feedback including subtask sequence + role assignment scheme.

[0090] Match the predefined reference motion segment according to the received information.

[0091] Generate robot coordination instructions according to the motion segment and send the instructions to the motion controller while receiving the execution feedback of the robot.

[0092] Determine whether dynamic adjustment is needed, if yes, re-execute the above process, if no, mark the task as completed.

[0093] 3. Working process of the motion controller: Receive robot coordination instructions including reference motion segment instructions.

[0094] Parse the instructions and perform semantic matching action segments.

[0095] Generate a physical constraint motion sequence matching the motion clip based on a dynamics model.

[0096] Drive the joint and end effector to perform the carrying motion.

[0097] Output natural and smooth motion to the robot execution unit to perform the carrying motion.

[0098] The embodiments of the present application can be applied in the following scenarios: Warehouse logistics: In a warehouse environment, multiple robots cooperate to carry goods, improving warehouse efficiency. For example, carrying large goods from warehouse A to warehouse B requires passing through a narrow passage. The system can automatically decompose tasks and assign roles to achieve efficient carrying.

[0099] Home service: In a home environment, robots cooperate to complete tasks such as carrying and organizing items. For example, carrying furniture from the living room to the bedroom requires passing through a narrow corridor. The system can automatically handle complex scenarios to achieve smooth carrying.

[0100] Industrial manufacturing: In a factory environment, robots cooperate to carry and assemble large equipment. For example, carrying large mechanical parts from the production line to the assembly area requires the system to automatically assign robot roles based on equipment characteristics and environmental conditions to optimize the carrying process.

[0101] Emergency rescue: In disaster sites, robots cooperate to carry rescue supplies to improve rescue efficiency. For example, carrying first aid supplies from the rescue center to the disaster area requires passing through a collapsed building passage. The system can automatically analyze the environment and develop an optimal carrying plan.

[0102] Large-scale events: In large-scale event venues, robots cooperate to carry stage equipment, props, etc. For example, carrying stage equipment from the backstage to the stage requires passing through a narrow passage. The system can automatically handle multi-robot cooperation to ensure safe equipment carrying.

[0103] The embodiments of the present application fully utilize the pre-training knowledge of large language models and do not need to fine-tune for specific carrying scenarios. When facing new tasks and new environments (such as carrying long boxes through narrow doorways, crossing steps, etc.), the system can still achieve effective task decomposition and role assignment through natural language input, effectively solving the scene transfer problem, and outperforming traditional methods that require individual design of reward functions or cooperation rules for each carrying task.

[0104] The storage and computing overhead of the embodiments of the present application is extremely low, and is compatible with existing robot systems. The reference motion dataset only needs to store the basic motion clip vector of general carrying skills, without storing high-dimensional scene-specific vectors. The collaborative decision-making module only calls the reference clip in real time during the task execution phase, significantly reducing system resource requirements and seamlessly integrating into existing robot control frameworks.

[0105] The embodiment of the present application has end-side friendliness. The method utilizes a general motion controller and a large language model reasoning capability deployed on the robot end side, guarantees privacy and security of the carrying operation data (sensitive task instructions do not need to be transmitted to the cloud), and has extremely low fusion computing overhead, does not cause additional significant load to the robot end side device, and supports real-time and efficient task execution.

[0106] A multi-agent collaboration method for humanoid robots corresponding to the above embodiment, Figure 6 A structural block diagram of a multi-agent collaboration system for humanoid robots is provided for an embodiment of the present application. Only parts related to the embodiments of the present application are shown for ease of illustration. Reference is made to Figure 6 The multi-agent collaboration system 20 for humanoid robots includes a task decomposition module 201, a collaborative decision module 202, and a motion control module 203. The task allocation module 201 is configured to: In response to receiving a natural language description of a target task, the target task is parsed to obtain a subtask sequence corresponding to the target task; Based on the type of each subtask in the subtask sequence, a structured subtask sequence and a robot role allocation strategy are determined, the structured subtask sequence includes a cooperation timing sequence of at least one execution robot, and the robot role allocation strategy includes at least one execution robot allocated for each subtask, the at least one execution robot being a humanoid robot with an embedded agent; The structured subtask sequence and the robot role allocation strategy are packaged to generate a task state; The collaborative decision module 202 is configured to: Align the task state and the environment feedback data to construct a joint vector; Retrieving at least one motion segment matching the joint vector from a pre-defined reference motion data set, and generating robot collaboration instructions based on the at least one motion segment, the pre-defined reference motion data set including a plurality of motion segments for controlling the execution of robot actions; The motion control module 203 is configured to: Control at least one execution robot to perform corresponding motions according to the robot collaboration instructions.

[0107] In an embodiment of the present application, the task allocation module 201 is specifically configured to: Based on a pre-determined structured prompt, the target task is semantically parsed to obtain a subtask list corresponding to the target task; the structured prompt is constructed based on a task context, a pre-defined action primitive library, and an output format constraint; The subtask list is parsed to extract a subtask sequence corresponding to the target task.

[0108] In one embodiment of this application, the collaborative decision-making module 202 is specifically used for: The joint vector is jointly encoded with each motion segment in the predefined reference motion dataset; and the adaptation scores of the joint vector and each motion segment are obtained based on the joint encoding, resulting in multiple matching scores. Based on the matching score, at least one motion segment that matches the joint vector is determined.

[0109] In one embodiment of this application, the motion control module 203 is specifically used for: Semantic parsing is performed on the robot's cooperative instructions to obtain the task description vector; Determine at least one best-matching action template for the robot based on the task description vector; Control at least one execution robot to perform the corresponding motion based on the best matching motion template of at least one execution robot.

[0110] In one embodiment of this application, the motion control module 203 is specifically used for: The task description vector is matched with the metadata labels in the predefined reference motion dataset to determine the best matching motion template for at least one executing robot.

[0111] In one embodiment of this application, the motion control module 203 is specifically used for: For each execution robot, a discrete motion sequence is generated based on the optimal motion template of that execution robot, and the execution robot is controlled to perform the corresponding motion based on the discrete motion sequence.

[0112] In one embodiment of this application, the collaborative decision-making module 202 is specifically used for: The task status and environmental feedback data are sequentially timestamped and their coordinates are uniformly transformed to obtain a joint vector.

[0113] See Figure 7 , Figure 7 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 7 The electronic device 300 shown in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304.

[0114] It should be understood that, in the embodiments of the present application, the processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0115] The input device 302 can include a touchpad, a fingerprint collection sensor (for collecting fingerprint information and direction information of a fingerprint of a user), a microphone, etc., and the output device 303 can include a display (LCD, etc.), a speaker, etc.

[0116] The memory 304 can include read-only memory and random access memory, and provide instructions and data for the processor 301. A portion of the memory 304 can also include non-volatile random access memory.

[0117] In specific implementations, the processor 301, the input device 302 and the output device 303 described in the embodiments of the present application can execute the implementation manners described in the multi-agent collaboration method for a humanoid robot provided by the embodiments of the present application, and can also execute the implementation manners of the electronic device described in the embodiments of the present application, which will not be described here.

[0118] In another embodiment of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program includes program instructions, and the program instructions are executed by a processor to implement all or part of the processes of the above-mentioned embodiment methods. The computer program can also be used to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer readable medium can include any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0119] The computer readable storage medium can be an internal storage unit of the electronic device of any of the preceding embodiments, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, or the like equipped on the electronic device. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the electronic device. The computer readable storage medium is used to store a computer program and other programs and data required by the electronic device. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0120] Those skilled in the art can understand that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic device and the units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0122] In several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces or units, and can also be electrical, mechanical or other forms of connection.

[0123] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0124] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-agent collaboration method for a humanoid robot, characterized by, The method comprises: in response to receiving a natural language description of a target task, parsing the target task to obtain a subtask sequence corresponding to the target task; determining a structured subtask sequence and a robot role allocation strategy based on the type of each subtask in the subtask sequence, the structured subtask sequence comprising a cooperation timing sequence of at least one execution robot, and the robot role allocation strategy comprising at least one execution robot allocated for each subtask, the at least one execution robot being a humanoid robot with an embedded agent; packaging the structured subtask sequence and the robot role allocation strategy to generate a task state; aligning the task state and environment feedback data to construct a joint vector; retrieving at least one motion segment matching the joint vector from a predefined reference motion dataset, and generating robot coordination instructions based on the at least one motion segment, the predefined reference motion dataset comprising a plurality of motion segments for controlling the actions of execution robots; controlling at least one execution robot to perform corresponding motions according to the robot coordination instructions.

2. The multi-agent collaboration method for a humanoid robot according to claim 1, wherein, The parsing of the target task to obtain a subtask sequence corresponding to the target task comprises: performing semantic analysis on the target task based on a predetermined structured prompt to obtain a subtask list corresponding to the target task; the structured prompt is constructed based on task context, a predefined action primitive library, and output format constraints; performing analysis on the subtask list to extract a subtask sequence corresponding to the target task.

3. The multi-agent collaboration method for a humanoid robot according to claim 1, wherein The retrieving of at least one motion segment matching the joint vector from a predefined reference motion dataset comprises: jointly encoding the joint vector and each motion segment in the predefined reference motion dataset; and obtaining a plurality of matching scores according to the adaptation scores of the joint vector and each motion segment obtained from the joint encoding; and determining at least one motion segment matching the joint vector according to the matching scores.

4. The multi-agent collaboration method for a humanoid robot according to claim 1, wherein The controlling of at least one execution robot to perform corresponding motions according to the robot coordination instructions comprises: performing semantic analysis on the robot coordination instructions to obtain a task description vector; determining a most matching action template of at least one execution robot based on the task description vector; controlling at least one execution robot to perform corresponding motions based on the most matching action template of at least one execution robot.

5. The multi-agent collaboration method for a humanoid robot according to claim 4, wherein, The determining of a most matching action template of at least one execution robot based on the task description vector comprises: matching the task description vector with metadata tags in a predefined reference motion dataset to determine a most matching action template of at least one execution robot.

6. The multi-agent collaboration method for a humanoid robot according to claim 4, wherein The controlling of at least one execution robot to perform corresponding motions based on the most matching action template of at least one execution robot comprises: for each execution robot, generating a discrete action sequence based on the best action template of the execution robot, and controlling the execution robot to perform corresponding motions based on the discrete action sequence.

7. The multi-agent collaboration method for a humanoid robot according to claim 1, wherein, The aligning of the task state and environment feedback data to construct a joint vector comprises: The task state and environment feedback data are sequentially timestamped, aligned and coordinate-uniformly converted to obtain a joint vector.

8. A multi-agent collaboration system for humanoid robots, characterized by, Comprise: Task decomposition module, collaborative decision module and motion control module; The task allocation module is used for: In response to receiving a natural language description of a target task, parsing the target task to obtain a subtask sequence corresponding to the target task; Based on the type of each subtask in the subtask sequence, determine a structured subtask sequence and a robot role allocation strategy, the structured subtask sequence includes the cooperation timing of at least one execution robot, and the robot role allocation strategy includes at least one execution robot allocated for each subtask, and the at least one execution robot is a humanoid robot with an embedded agent; The structured subtask sequence and the robot role allocation strategy are packaged to generate a task state; The collaborative decision module is used for: Align the task state and environment feedback data to construct a joint vector; Retrieving at least one motion segment matched with the joint vector from a pre-defined reference motion data set, and generating a robot collaborative instruction based on the at least one motion segment, the pre-defined reference motion data set including a plurality of motion segments for controlling the action of an execution robot; The motion control module is used for: According to the robot collaborative instruction, control at least one execution robot to execute corresponding motion.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 7.

Images