Heterogeneous multi-machine cooperation method, terminal and readable storage medium

Abstract

The invention relates to the technical field of robots, and discloses a heterogeneous multi-machine cooperation method, a terminal and a readable storage medium, and the method comprises the steps: obtaining environment perception data, and obtaining a multi-modal fusion scene prompt according to the environment perception data; distributing a role for each agent according to the multi-modal fusion scene prompt, and generating an action plan of each agent according to the multi-modal fusion scene prompt and the role of each agent; and controlling each agent to execute the respective corresponding action plan. According to the method, the multi-modal fusion scene prompt is extracted according to the environment perception data, the parameters of different scenes are matched according to the multi-modal fusion scene prompt, the role of each agent is allocated through the role allocation algorithm, and the method does not depend on predefined roles and the parameters specific to the scenes; and the task execution capability of the system in a dynamic scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and particularly to a heterogeneous multi-robot cooperation method, a terminal, and a readable storage medium. Background Art

[0002] With the rapid development of robot technology, for large-scale tasks in dangerous environments, robot systems can prevent humans from being directly exposed to potentially dangerous environments. However, in the actual task environment, the dynamic and unpredictable nature of the world makes it difficult to design an autonomous robot that can effectively adapt to all environments.

[0003] In view of this situation, heterogeneous multi-robot systems have received extensive attention and applications due to their unique advantages. Heterogeneous multi-robot systems include robots with various shapes, sizes, and capabilities, such as unmanned aerial vehicles (UAVs), unmanned ground vehicles, etc. UAVs have a wide field of view and the ability to ignore terrain, but their own load capacity is poor; unmanned ground vehicles have a large load but cannot perform large-scale environmental perception. The heterogeneous system composed of UAVs and ground vehicles can complement each other well, taking into account the wide field of view brought by the aerial UAVs and the high load capacity of ground robots. The UAV provides a broader field of view for the formation of ground vehicles, helping the ground vehicles to perform trajectory planning over a larger range, so as to avoid obstacles in advance; the unmanned vehicle acts as a ground base station to ensure the effective communication of the UAV, and at the same time carries advanced sensors to cooperate and fill in detailed information.

[0004] Based on this, in order to complete a specific task, robots with various shapes, sizes, and capabilities need to cooperate with each other. Therefore, it is necessary to provide a heterogeneous multi-agent system (HMAS) as a key component for performing complex tasks in dynamic and uncertain environments. However, existing heterogeneous multi-agent systems (HMAS) are often limited by predefined roles and scenario-specific parameters, which weakens their flexibility and scalability, resulting in the difficulty of HMAS to effectively adapt to real-time environmental changes, and further hindering the ability to perform tasks efficiently under different conditions. Summary of the Invention

[0005] The purpose of the present invention is to provide a heterogeneous multi-robot cooperation method, a terminal, and a readable storage medium, aiming to solve the problem that existing heterogeneous multi-agent systems are limited by predefined roles and scenarios and have low flexibility.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] The present invention provides a heterogeneous multi-robot cooperation method, and the heterogeneous multi-robot cooperation method includes:

[0008] Obtain environmental perception data, and obtain multi-modal fusion scenario prompts according to the environmental perception data;

[0009] Allocate roles to each agent according to the multi-modal fusion scenario prompts, and generate action plans for each agent according to the multi-modal fusion scenario prompts and the roles of each agent;

[0010] Control each agent to execute its corresponding action plan.

[0011] Furthermore, the environmental perception data includes RGB images and lidar data;

[0012] The obtaining of the multi-modal fusion scenario prompts according to the environmental perception data specifically includes:

[0013] Obtain instance segmentation results and semantic descriptions according to the RGB images;

[0014] Obtain point cloud features according to the lidar data;

[0015] Connect the point cloud features, the instance segmentation results and the semantic descriptions to obtain the multi-modal fusion scenario prompts.

[0016] Furthermore, the allocating of roles to each agent according to the multi-modal fusion scenario prompts specifically includes:

[0017] Obtain scene tasks, decompose the scene tasks into multiple subtasks, and divide agents into each subtask;

[0018] For each subtask, obtain all roles of each subtask;

[0019] For each subtask, allocate roles to each agent according to the multi-modal fusion scenario prompts.

[0020] Furthermore, the allocating of roles to each agent according to the multi-modal fusion scenario prompts is specifically:

[0021] Directly match the attributes of the agent with the subtask requirements according to the multi-modal fusion scenario prompts:

[0022]

[0023] where Match is a function that measures the degree of matching of the agent's attributes, scene description features, and environmental object characteristics with the requirements of a specific role r, R represents the set of all possible roles, A i represents the attributes of the i-th agent, represents the scene description features at time t, O(t) represents the environmental object characteristics at time t, r i(t) represents the role of the i-th agent at time t.

[0024] Further, the step of allocating each agent into each of the roles specifically includes:

[0025] Obtain the role allocation, iteratively optimize the loss value of the role allocation, and obtain the optimal role allocation with the lowest loss value;

[0026] Allocate roles to each agent according to the optimal role allocation.

[0027] Further, the loss value specifically includes the deviation cost between the role allocation and the task requirements, the execution efficiency cost, and the conflict cost between the role allocation and the environmental safety.

[0028] Further, the heterogeneous multi-machine cooperation method further includes:

[0029] When controlling each agent to execute its corresponding action plan, continuously monitor the trajectories and postures of each agent;

[0030] If the trajectory error of an agent is greater than the set trajectory error threshold or the posture error is greater than the set posture error threshold, mark it as abnormal and regenerate the action plan.

[0031] Further, the heterogeneous multi-machine cooperation method further includes:

[0032] When controlling each agent to execute its corresponding action plan, continuously obtain the environmental perception data and detect whether the environmental perception data changes;

[0033] If the environmental perception data changes, regenerate the action plan.

[0034] In addition, to achieve the above object, the present invention also provides a terminal, the terminal includes: a memory, a processor, and a heterogeneous multi-machine cooperation program stored on the memory and executable on the processor, and when the heterogeneous multi-machine cooperation program is executed by the processor, it controls the terminal to implement the steps of the above-mentioned heterogeneous multi-machine cooperation method.

[0035] In addition, to achieve the above object, the present invention also provides a readable storage medium, the readable storage medium stores a heterogeneous multi-machine cooperation program, and when the heterogeneous multi-machine cooperation program is executed by a processor, it implements the steps of the above-mentioned heterogeneous multi-machine cooperation method.

[0036] The present invention adopts the above technical solutions and has the following effects:

[0037] The present invention extracts multi-modal fusion scene cues from environmental perception data, matches parameters for different scenes according to the multi-modal fusion scene cues, and assigns roles to each agent through a role assignment algorithm, without relying on predefined roles and scene-specific parameters, thereby improving the system's ability to execute tasks in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flowchart of the steps of a heterogeneous multi-machine cooperation method in a preferred embodiment of the present invention;

[0039] Figure 2 is an automatic flowchart of a heterogeneous multi-machine cooperation method in a preferred embodiment of the present invention;

[0040] Figure 3 is a stage flowchart of a heterogeneous multi-machine cooperation method in a preferred embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not used to limit the present invention.

[0043] Embodiment 1

[0044] Specifically, please refer to Figure 1 、 Figure 2 and Figure 3 , Embodiment 1 of the present application is a heterogeneous multi-machine cooperation method, which includes the steps of:

[0045] S1. Obtain environmental perception data, and obtain multi-modal fusion scene cues according to the environmental perception data.

[0046] Specifically, in this embodiment, environmental perception is realized by collecting multi-modal sensor data and motion perception data to obtain environmental perception data, and the environmental perception data includes RGB images and lidar (LiDAR) data.

[0047] These environmental perception data provide information for assisting task decision-making. For example, in a logistics scenario, they provide the basic information required to understand the scene, identify the positions of objects, and detect obstacles. The present invention uses these data to perceive its environment in real time and can make an adaptive response to any dynamic changes, such as a suddenly appearing obstacle in the path.

[0048] After that, the present invention obtains a multi-modal fusion scene prompt based on the environmental perception data, where the multi-modal fusion scene prompt includes scene description features, segmentation semantic features, and point cloud features.

[0049] Specifically, at time t, the RGB image is input into an instance segmentation model to extract the boundaries and semantic classifications of objects in the image, obtaining an instance segmentation result S InstSeg (t). The RGB image is input into a vision-language model, and the vision-language model generates high-level semantic information, including the classification, location, and relationships of scene objects as the semantic description S RGB (t). The point cloud features S LiDAR (t) are extracted according to the LiDAR data. Then, through the Concat (concatenation) function, the instance segmentation result S InstSeg (t), the semantic description S RGB (t), and the point cloud features S LiDAR (t) are fused to obtain the scene description features at time t.

[0050] S2. Assign roles to each agent according to the multi-modal fusion scene prompt, and generate action plans for each agent according to the multi-modal fusion scene prompt and the roles of each agent.

[0051] Specifically, the present invention first uses the task decomposition function of a large language model (LLM) to decompose complex scene tasks into smaller and more manageable subtasks:

[0052]

[0053] Among them, T(t) represents the subtask at time t, G(t) represents the scene task at time t, Δ represents the task decomposition function. Then, according to the capabilities of each agent and the characteristics of the environmental objects involved in the subtasks, each agent is assigned to different subtasks. When dividing, the priorities of each subtask need to be considered to ensure that tasks with high priorities can be completed first.

[0054] For example, in a logistics task, the task of transporting items from a warehouse to a distribution center can be decomposed into subtasks such as selecting the best transportation route, inspecting the vehicle status, and ensuring a successful delivery by avoiding obstacles.

[0055] Then, for each subtask, the roles in the current task are obtained, and then a role assignment function is used to assign a role to each agent:

[0056]

[0057] Among them, r i (t) represents the role of the i-th agent at time t, γrole Denotes the role assignment function, A i Denotes the attributes of the i-th agent, and O(t) denotes the characteristics of the environmental objects at time t.

[0058] Specifically, the role assignment function γ role Can be a rule-based role assignment function or an optimization-based role assignment model function.

[0059] If it is a rule-based role assignment function, it directly matches according to the attributes of the agent and the task requirements:

[0060]

[0061] Among them, Match is a rule-based role assignment function, specifically a function that measures the degree of matching of the agent attributes, scene description features, and environmental object characteristics with the requirements of a specific role r, and R represents the set of all possible roles.

[0062] If it is an optimization-based role assignment model function, the loss function is:

[0063]

[0064] Among them, L represents the loss value, C task Denotes the deviation cost function between role assignment and task requirements, C efficiency Denotes the execution efficiency cost function, C safety Denotes the conflict cost between role assignment and environmental safety. The optimization-based role assignment model function optimizes role assignment by minimizing the loss value.

[0065] After completing the role assignment, what needs to be done is more specific task planning. For example, in a logistics task, the subtasks include determining the best transportation route, allocating inspection tasks to ensure route safety, and coordinating search and rescue tasks in case of lost items. For each subtask, path planning is performed using 3D coordinates and a time planning header to ensure respect for spatial and time constraints, and obstacle avoidance and route optimization algorithms (such as PID or NMPC) are applied to ensure safe and efficient transportation.

[0066] By improving the task allocation model, the present invention introduces large multi-modal models (VLM and LLM) to achieve dynamic task decomposition and role self-organization. The system decomposes complex tasks into sub-tasks that can be executed in parallel through the fusion of multi-modal data (such as RGB, InstanceSeg, and LiDAR), combined with intelligent task scheduling methods. At the same time, the central planner dynamically generates task allocations and schedules based on user instructions and adaptively adjusts roles in the scenario. Compared with traditional methods, this improvement significantly enhances the scalability and task adaptability of the system. For example, in the inspection scenario, the system has achieved a reduction in task completion time to 14.67 seconds and an increase in success rate to 83.31%. This multi-modal self-organization framework surpasses fixed task architectures and has the ability to adjust tasks across scenarios.

[0067] S3. Control each of the agents to execute their respective corresponding action plans.

[0068] Specifically, control each of the agents to execute tasks in real time in various operation scenarios such as logistics, inspection, and search and rescue. In the logistics scenario, the system monitors the transportation status and reports the task progress and task completion rate. In the inspection scenario, the agent evaluates the status of equipment on the transportation route and reports the progress and completion rate. The search and rescue scenario focuses on locating and retrieving target items and reports task status and completion updates accordingly.

[0069] During the process of controlling each of the agents to execute their respective corresponding action plans, at each time step, the system executes tasks according to the updated planned path and repeats the loop until the task is completed. During the process of each agent executing the task, as the environment develops, the system will detect changes, such as changes in the position of obstacles or changes in wind speed. The environmental feedback will inform the system of these changes, which may lead to errors in task execution. In response, a self-correction mechanism will be triggered, enabling the system to recalculate the path and adjust the task plan to ensure that the operation continues to be effective. This continuous feedback loop allows for dynamic optimization of task execution.

[0070] In addition, the present invention also monitors anomalies in the task process. At time t, obtain the expected trajectory p desired,i (t) of the i-th agent, and the actual trajectory p i (t) of the i-th agent, and calculate the difference between the two to obtain the trajectory error Δp i (t):

[0071] Δp i (t) = p desired,i (t) - p i (t);

[0072] And, at time t, obtain the expected attitude q desired,i(t) and the actual attitude q of the i-th agent i (t), and then according to the desired attitude q of the i-th agent desired,i (t) and the actual attitude q of the i-th agent i (t) to calculate the attitude error Δq i (t):

[0073]

[0074] If the trajectory error Δp i (t) is greater than the set trajectory error threshold ∈ att or the attitude error Δq i (t) is greater than the set attitude error threshold ∈ pos , then mark the i-th agent as abnormal and task adjustment and control update are required.

[0075] When an anomaly is detected, the large language model re-plans the task and updates the control instructions to adjust the behavior of the agent to ensure that the task can be correctly executed.

[0076] In terms of the improvement of the self-correction mechanism and environmental perception, the present invention designs a two-layer self-correction mechanism to reduce latency and reduce human intervention. The system adjusts the global feedback according to multi-modal inputs, and realizes high-precision environmental perception and task optimization by constructing a unified scene description model. The system uses multi-modal data to generate a fused scene description, and realizes real-time feedback and dynamic adjustment of task execution through the self-correction mechanism. Through the path planning optimization supported by VLM and LLM, this mechanism can quickly identify anomalies and correct deviations in complex environments. Especially in search and rescue scenarios, the system shows the ability to accurately perceive and avoid dynamic obstacles, and the success rate is increased to 69.53%, and the task completion time is shortened to 18.56 seconds. The cloud-edge collaborative feedback loop improves the system robustness and resilience, ensuring the safe and efficient operation of multi-agent tasks in high-risk environments.

[0077] In addition, the present invention also aims to solve the problem that the advantages of systems based on large language models (LLMs) are not fully utilized in environments that require advanced reasoning and task decomposition. By deploying large models at the central node, the present invention optimizes the task allocation and complex decision-making processes, and brings the capabilities of low-level perception and agile computing to the edge to alleviate the computing challenges and achieve more efficient and responsive multi-agent operations in dynamic environments. This hybrid approach balances the computing load between the central node and the edge node, enhancing the overall performance and adaptability of the system in real-time applications.

[0078] The method of the present invention is tested based on logistics scenarios, inspection scenarios, and rescue scenarios. Based on five key performance indicators, the performance of the system across multiple task scenarios is comprehensively evaluated. The indicators include task success rate, task execution steps, task completion time, adaptability, and scalability.

[0079] Among them, the task success rate indicator measures the reliability of the system to successfully complete a specific task. In the experiment, the system runs 100 times under different initial conditions and random environmental variables in each scenario to simulate the uncertainty of the real world, and the number of times of successfully completing the task is recorded to calculate the success rate.

[0080] The task execution steps indicator measures the average number of operation steps required for the system to complete a single task, including subtasks such as information processing, path planning, and execution. By repeating the task multiple times, the steps required for each execution are recorded, and the average task execution steps in different scenarios are calculated.

[0081] The task completion time indicator measures the total time (in seconds) from the start of the task to the successful completion of the system, reflecting the overall efficiency of the system. In each scenario, the completion time is recorded in multiple tests, and the average time is used to evaluate the responsiveness of the system.

[0082] The adaptability indicator measures the self-adjustment ability of the system to dynamic environmental changes (e.g., the appearance of obstacles or changes in the target location). During the experiment, sudden changes in the environment are introduced to observe whether the system can quickly adjust its strategy and effectively complete the task.

[0083] The scalability indicator evaluates the performance stability of the system under extended task complexity or multi-agent collaboration. This metric tests how the system handles increased task complexity or concurrent tasks and evaluates the performance of the system in high-load scenarios to quantify its scalability.

[0084] In the logistics scenario, under different initial conditions and random environmental variables, by running 100 times to simulate the uncertainty of the real world, the task success rate obtained by the present invention reaches 80.18%, the task execution steps are as low as 8.12 steps, the task completion time is reduced to 16.84 seconds, and the adaptability and scalability are also significantly better than the prior art.

[0085] For the parameter indicators of more scenarios, please refer to Table 1:

[0086] Table 1: Parameter comparison table

[0087]

[0088] Among them, IC3Net, FAVMAC, and SMRC-LLM are heterogeneous multi-agent systems of the prior art.

[0089] Example Two

[0090] Please refer to Figure 4 , based on the above method, the present invention further provides a terminal, which includes a processor 10, a memory 20, and a display 30. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively.

[0091] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In some other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as the program code for installing the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a heterogeneous multi-machine cooperation program 40 is stored on the memory 20, and this heterogeneous multi-machine cooperation program 40 can be executed by the processor 10 to implement the terminal in the present application.

[0092] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chips, and is used to run the program code stored in the memory 20 or process data, such as executing the relevant programs of the above-mentioned heterogeneous multi-machine cooperation method.

[0093] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light Emitting Diode) toucher, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0094] In one embodiment, when the processor 10 executes the heterogeneous multi-machine cooperation program 40 in the memory 20, the steps of the above-mentioned heterogeneous multi-machine cooperation method are implemented.

[0095] Example Three

[0096] This embodiment provides a storage medium. The readable storage medium stores a heterogeneous multi-machine cooperation program. When the heterogeneous multi-machine cooperation method program is executed by a processor, it implements the steps of the above-mentioned heterogeneous multi-machine cooperation.

[0097] In summary, the present invention extracts multi-modal fusion scene cues through environment perception data, adapts parameters for different scenarios according to the multi-modal fusion scene cues, and assigns roles to each agent through a role assignment algorithm. Without relying on predefined roles and scenario-specific parameters, it improves the system's ability to execute tasks in dynamic scenarios.

[0098] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or terminal. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or terminal including that element.

[0099] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above method embodiments. The storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0100] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A heterogeneous multi-machine collaboration method, characterized in that: The heterogeneous multi-machine collaboration method comprises: Acquire environmental perception data, and acquire multimodal fusion scene prompts according to the environmental perception data; Assigning a role to each intelligent agent according to the multimodal fusion scene prompt, and generating an action plan for each intelligent agent according to the multimodal fusion scene prompt and the role of each intelligent agent; Control each of the intelligent agents to execute the corresponding action plan.

2. A heterogeneous multi-machine collaboration method according to claim 1, characterized in that: The environmental perception data includes RGB images and laser radar data; The acquiring of a multimodal fusion scene prompt according to the environmental perception data specifically includes: Acquire instance segmentation results and semantic descriptions according to the RGB image; Acquire point cloud features according to the laser radar data; The point cloud features, the instance segmentation results and the semantic description are connected to obtain the multimodal fusion scene prompt.

3. A heterogeneous multi-machine collaboration method according to claim 1, characterized in that: The assigning a role to each agent according to the multimodal fusion scene prompt specifically includes: Obtain a scenario task, decompose the scenario task into multiple subtasks, and divide the agent into each of the subtasks; For each of the subtasks, obtain all roles of each of the subtasks; For each subtask, a role is assigned to each agent according to the multimodal fusion scenario prompts.

4. A heterogeneous multi-machine collaboration method according to claim 3, characterized in that: The assigning of roles to each agent according to the multimodal fusion scene prompt is specifically: According to the multimodal fusion scenario prompts, the attributes of the agent are directly matched with the subtask requirements: Among them, Match is a function that measures the degree of match between the agent attributes, scene description features, and environmental object characteristics and the requirements of a specific role r. R represents the set of all possible roles, and A i represents the attributes of the ith agent, represents the scene description feature at time t, O(t) represents the environmental object characteristics at time t, and r i (t) represents the role of the i-th agent at time t.

5. A heterogeneous multi-machine collaboration method according to claim 3, characterized in that: The allocating each agent to each role specifically includes: Obtain role allocation, iteratively optimize the loss value of role allocation, and obtain the optimal role allocation with the lowest loss value; Assign a role to each agent according to the optimal role assignment.

6. A heterogeneous multi-machine collaboration method according to claim 5, characterized in that: The loss value specifically includes the deviation cost between role allocation and task requirements, execution efficiency cost, and conflict cost between role allocation and environmental safety.

7. A heterogeneous multi-machine collaboration method according to claim 1, characterized in that: The heterogeneous multi-machine collaboration method further includes: When controlling each of the intelligent agents to execute the corresponding action plan, continuously monitoring the trajectory and posture of each of the intelligent agents; If the trajectory error of an agent is greater than the set trajectory error threshold or the posture error is greater than the set posture error threshold, the abnormality is marked and the action plan is regenerated.

8. The heterogeneous multi-machine collaboration method according to claim 1, characterized in that: The heterogeneous multi-machine collaboration method further includes: When controlling each of the intelligent agents to execute the corresponding action plans, continuously acquiring the environmental perception data and detecting whether the environmental perception data changes; If the environmental perception data changes, the action plan is regenerated.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a heterogeneous multi-machine collaboration method program stored in the memory and executable on the processor. When the heterogeneous multi-machine collaboration method program is executed by the processor, the terminal is controlled to implement the steps of a heterogeneous multi-machine collaboration method as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores a heterogeneous multi-machine collaboration program, which, when executed by a processor, implements the steps of a heterogeneous multi-machine collaboration method as described in any one of claims 1 to 8.

Images