Multi-robot cooperation system based on cloud machine hybrid intelligent driving

Through the multi-robot collaborative system driven by cloud machine hybrid intelligence, the problem of uncertainty and learning difficulties of the multi-robot system in complex scenarios is solved, closed-loop process innovation and efficient collaborative operations are achieved, and the overall efficiency and stability of the system are improved. sex.

CN118938904APending Publication Date: 2024-11-12NANKAI UNIV
View PDF 0 Cites 8 Cited by

Patent Information

Application Number
CN202410982195.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

There are uncertainty and learning difficulties in collaborative tasks in existing multi-robot systems in complex scenarios, and lack effective unified collaborative scheduling methods, making it difficult to achieve the desired collaborative effect in an asynchronous environment. Research on the combination of cloud computing power and robot terminals Fewer, resulting in insufficient efficiency and robustness.

Method used

Provide a multi-robot collaborative system based on cloud machine hybrid intelligent drive, including multi-robot task scheduling and wireless communication protocol module, multi-robot path planning and obstacle avoidance module and intelligent factory map modeling module, through cloud intelligent analysis and processing and robot real-time Perception, realize task decomposition, allocation, path planning and obstacle avoidance, build a closed-loop control system, and optimize task execution and environmental modeling.

Benefits of technology

Achieve closed-loop process innovation of multi-robot systems, improve the overall efficiency and stability of collaborative operations, reduce development costs, enhance the embodied capabilities of cloud intelligence and the scalability of systems, and adapt to efficient collaborative operations in complex environments. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938904A_ABST
    Figure CN118938904A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-robot cooperative system based on cloud machine hybrid intelligent driving. The multi-robot cooperative system comprises a multi-robot task scheduling and wireless communication protocol module, a multi-robot path planning and obstacle avoidance module and an intelligent factory map modeling module. And the multi-robot task scheduling and wireless communication protocol module is used for decomposing and distributing complex tasks and realizing communication between cloud machines. The intelligent factory map modeling module receives existing video data in the intelligent manufacturing environment, real-time environment map modeling is carried out on the intelligent manufacturing environment through image distortion correction, obstacle segmentation and splicing of a plurality of local images, and a global environment map is generated. And the multi-robot path planning and obstacle avoidance module receives the global environment map generated by the intelligent factory map modeling module and each robot subtask of the multi-robot task scheduling and wireless communication protocol module, generates an initial path for each robot, and detects and dynamically adjusts the motion path of the robot in real time in the task execution process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a multi-robot collaborative system based on cloud-machine hybrid intelligent drive. Background Art

[0002] At present, various types of robots play a very important role in various fields such as manufacturing, transportation, and life, and have shown great development potential in terms of completion efficiency and comprehensive cost. If the robots can be coordinated to work together, it will be of great significance to improve the intelligence of the robot group.

[0003] At present, the research on multi-robot systems has made initial progress, but there are still certain technical challenges in how to deeply integrate multi-robot systems with manufacturing and the real economy to promote the transformation and upgrading of the intelligent manufacturing field. On the one hand, the existing multi-robot system lacks an effective and unified collaborative scheduling method. In complex scenarios such as intelligent manufacturing workshops with dynamically changing communication topologies, environmental uncertainties, and nonlinear characteristics, the existing control strategies make it difficult for multi-agent systems to achieve the desired collaborative effects in an asynchronous environment. On the other hand, the collaborative tasks of multi-robot systems in complex scenarios have higher uncertainty and learning difficulties, and the limited computing resources of existing computer terminals make it difficult to meet the real-time and effectiveness of task execution.

[0004] In order to solve the technical problems existing in the above multi-robot system, researchers at home and abroad have optimized the strategy algorithm, network model, credit allocation and other aspects, and verified the effectiveness of the theoretical method through simulation experiments and practical applications. Although these research methods have improved the collaborative ability of the multi-robot system to a certain extent and accelerated the skill learning speed of the multi-robot system, most of these methods are carried out for robot terminals. Although they are helpful to complete the task, the required hardware upgrades greatly increase the cost of development and maintenance. In addition, most of these methods only start from the work efficiency of the robot to complete the task, and only consider the single task execution process, without considering the overall task decomposition, specific task allocation, and execution of the global workflow of closed-loop feedback. At present, there are few reports on the relevant research results of combining the powerful computing power of the cloud with the robot terminal to make up for the intelligence defects of the robot itself and improve the efficiency and robustness of the multi-robot system. Therefore, the multi-robot system driven by cloud-machine hybrid intelligence urgently needs further in-depth research. Summary of the invention

[0005] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to provide a multi-robot collaborative system based on cloud-machine hybrid intelligent drive.

[0006] The technical solution of the present invention to solve the technical problem is to provide a multi-robot collaborative system based on cloud-machine hybrid intelligent drive, characterized in that the system includes a multi-robot task scheduling and wireless communication protocol module, a multi-robot path planning and obstacle avoidance module and an intelligent factory map modeling module;

[0007] The multi-robot task scheduling and wireless communication protocol module is used to decompose and allocate complex tasks and realize communication between cloud machines. At the same time, it provides the multi-robot path planning and obstacle avoidance module with information on the subtasks and position relationships of each robot. When the task environment or task requirements change dynamically, it receives feedback information from the multi-robot path planning and obstacle avoidance module to re-plan the robot's motion path.

[0008] The intelligent factory map modeling module receives the existing video data in the intelligent manufacturing environment, and performs real-time environmental map modeling of the intelligent manufacturing environment through image distortion correction, obstacle segmentation and multiple local image stitching to generate a global environmental map. At the same time, it provides the necessary global map and environmental information for the multi-robot path planning and obstacle avoidance module, helping the system to accurately locate the robot position to plan the task path, and update the global environmental map in real time according to the movement changes of the multi-robot path planning and obstacle avoidance module;

[0009] The multi-robot path planning and obstacle avoidance module receives the global environment map generated by the intelligent factory map modeling module and the robot subtasks of the multi-robot task scheduling and wireless communication protocol module, generates an initial path for each robot, and detects and dynamically adjusts the robot's motion path in real time during task execution. That is, when the motion path changes dynamically, the robot will provide feedback to the cloud intelligence and request to re-plan the path. At the same time, before receiving the new path sent back by the cloud intelligence, the robot will make local path adjustments based on its own intelligence to achieve autonomous obstacle avoidance, avoid collisions with other robots and other safety hazards.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] (1) Closed-loop process innovation for multi-robot collaborative operations: Current research on multi-robot systems is mainly focused on the task execution stage, and is committed to improving the quality and efficiency of task execution. However, there is still insufficient research on the early decomposition and allocation of tasks and the feedback links during the execution process, which limits the in-depth realization of automated control. The present invention proposes a comprehensive task execution mechanism that covers the entire process from task receipt to completion, aiming to build a closed-loop intelligent manufacturing process and achieve process integrity. This mechanism incorporates the task decomposition, allocation and feedback links into a closed-loop control system, thereby achieving deep integration of automated control and improving the intelligence level of multi-robot systems. It is of great significance to promote the development of multi-robot collaborative operations in the field of intelligent manufacturing.

[0012] (2) Multi-aspect optimization of the working process of multi-robot system: Make up for the shortcomings of the existing multi-robot system in terms of task scheduling, environment modeling, path planning, and cloud-machine communication. In terms of task scheduling, the semantic understanding and task planning capabilities of the generative large language model are used to decompose and allocate tasks. At the same time, a real-time and reliable robot-oriented communication protocol is designed to break the "communication barriers" between heterogeneous robots, so that heterogeneous robot systems can complete tasks collaboratively and efficiently. In terms of environment modeling, the existing video equipment in the intelligent manufacturing environment is used to reduce the cost of system development. At the same time, an obstacle contour calibration method is designed to accurately distinguish between feasible paths and infeasible obstacle areas in the intelligent manufacturing environment, reduce the impact of image distortion caused by sensor errors and optical characteristics, and improve the accuracy of map modeling. In terms of path planning, a path planning algorithm combining global and local is designed by comprehensively considering the path quality and time cost. At the beginning of the task, the cloud intelligence plans the initial path for the robot terminal. When encountering dynamically changing obstacles during the task execution, the robot terminal dynamically adjusts the motion path according to the local environment and requests the cloud intelligence to regenerate the path to ensure the safety and reliability of the robot movement.

[0013] (3) Constructing a new control system with complementary resource advantages: This invention creatively integrates the powerful processing power of cloud computing and the robot's own intelligence to form a new control system with complementary resource advantages. In a multi-robot system, through intelligent analysis and processing in the cloud, accurate task allocation and decision support are provided to the robot, which greatly reduces the real-time computing burden of the robot in a complex environment and improves the overall operating efficiency and stability of the system. This method innovatively realizes the optimal configuration of distributed intelligent resources and provides a new research perspective and technical path for the efficient collaborative operation of multi-robot systems in complex environments.

[0014] (4) Propose an innovative solution for the embodiment of cloud-based intelligence: Although cloud-based intelligence has powerful computing power support, it may lead to decision-making risks and efficiency risks due to the lack of true understanding of the task world. The present invention designs an adaptive active learning mechanism for cloud-based intelligence, using the multi-robot system as the "sensory extension" of cloud-based intelligence to establish a comprehensive and three-dimensional perception of the environment. This approach effectively breaks through the passivity of cloud-based intelligence in traditional supervised learning, reduces dependence on large-scale labeled data, realizes the embodied cognitive ability of cloud-based intelligence during task execution, and significantly improves the execution efficiency and robustness of cloud-based intelligence in complex task environments, providing a new theoretical framework and practical path for the embodiment research of cloud-based intelligence.

[0015] (5) High scalability of the system: The present invention is committed to building a multi-robot collaborative system that adapts to actual production needs. The system adopts a modular design concept and encapsulates the functional interfaces such as scheduling, communication, perception and decision-making required for multi-robot collaborative operations. This design makes the system architecture logically clear and hierarchical, which is convenient for interaction with the production process execution management system of the manufacturing enterprise in the smart factory. In addition, the design of the system can also adapt to other task scenarios such as smart logistics and smart warehousing, demonstrating its cross-domain application potential. In addition, the design of the system also takes into account possible future technological upgrades and functional expansions, providing a solid foundation for the long-term sustainable development and optimization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall structure of the system of the present invention;

[0017] Figure 2 It is a flow chart of the multi-robot task scheduling and wireless communication protocol module of the present invention;

[0018] Figure 3 It is a flow chart of the multi-robot path planning and obstacle avoidance module of the present invention;

[0019] Figure 4 It is a flow chart of the intelligent factory map modeling module of the present invention.

[0020] Multi-robot task scheduling and wireless communication protocol module 10, multi-robot path planning and obstacle avoidance module 20, smart factory map modeling module 30;

[0021] Communication protocol 101, message structure 1011, message flow 1012, message transmission mechanism 1013;

[0022] Task decomposition 102, natural language instructions 1021, large language model 1022, subtask sequence 1023;

[0023] Task allocation 103, task requirements 1031, role functions 1032, expert language model 1033, and robot subtasks 1034;

[0024] Cloud-based robot path planning 201, environmental status 2011, cloud-based intelligence 2012, initial path 2013, path optimization 2014;

[0025] Robot-based collision detection and obstacle avoidance 202, target recognition 2021, trajectory prediction 2022, collision analysis 2023, autonomous obstacle avoidance 2024, local path adjustment 2025;

[0026] Cloud-based active learning 203, multi-robot system 2031, environmental information 2032, embodied intelligence 2033;

[0027] Image distortion correction 301, video image 3011, distortion correction algorithm 3012, corrected image 3013;

[0028] Obstacle segmentation 302, semantic segmentation algorithm 3021, area map 3022;

[0029] Multi-camera image stitching 303, feature matching algorithm 3031, global environment map 3032. DETAILED DESCRIPTION

[0030] The specific embodiments of the present invention are given below. The specific embodiments are only used to further illustrate the present invention in detail and do not limit the protection scope of the present invention.

[0031] The present invention provides a multi-robot collaborative system based on cloud-machine hybrid intelligent drive (referred to as system), characterized in that the system includes a multi-robot task scheduling and wireless communication protocol module 10, a multi-robot path planning and obstacle avoidance module 20 and an intelligent factory map modeling module 30;

[0032] The multi-robot task scheduling and wireless communication protocol module 10 is the interface for the system to interact with the intelligent manufacturing platform. It is used to decompose and allocate complex tasks and realize communication between cloud machines. At the same time, it provides the multi-robot path planning and obstacle avoidance module 20 with information on the subtasks and position relationships of each robot. When the task environment or task requirements change dynamically, it receives feedback information from the multi-robot path planning and obstacle avoidance module 20 to replan the movement path of the robot.

[0033] Preferably, the multi-robot task scheduling and wireless communication protocol module 10 includes a communication protocol 101, task decomposition 102 and task allocation 103; after receiving the total task in natural language form sent by the existing manufacturing execution system of the factory, the cloud intelligence performs task decomposition 102 on the total task, and uses the communication protocol 101 between the cloud intelligence and the heterogeneous multi-robots to perform task allocation 103;

[0034] Communication protocol 101 is a universal communication protocol between cloud intelligence and heterogeneous multi-robots, which enables real-time command interaction and feedback between robots and servers;

[0035] Task decomposition 102 receives the general task in natural language form sent by the existing manufacturing execution system of the factory, and uses the semantic understanding ability and task planning ability of the large language model 1022 to decompose the complex task into several executable subtasks and generate a subtask sequence 1023;

[0036] Task assignment 103 trains the existing large language model 1022, comprehensively considers the nature of the subtasks, the functions of different robots and environmental information, and assigns the generated robot subtasks 1034 to the corresponding robots through the communication protocol 101.

[0037] Preferably, the process of multi-robot task scheduling and wireless communication protocol module 10 is as follows: Figure 2 As shown, specifically: first, the cloud-side intelligence receives the total task in the natural language form sent by the manufacturing execution system, sends the natural language instruction 1021 in the total task to the large language model 1022, and uses the semantic understanding ability and task planning ability of the large language model 1022 to generate a robot-executable sub-task sequence 1023; then the cloud-side intelligence sends the task requirements 1031, the robot's role function 1032 and location information as prior knowledge together with the sub-task sequence 1023 to the expert large language model 1033, so that it generates each robot sub-task 1034; finally, each robot sub-task 1034 is packaged according to the requirements of the message structure 1011, the message flow 1012 and the message transmission mechanism 1013 and sent to the corresponding robot.

[0038] Preferably, the semantic understanding ability and task planning ability of the large language model 1022 are used to generate a robot-executable subtask sequence 1023, specifically: the cloud-based intelligence constructs a task decomposition data set in a bottom-up manner, and uses subtasks that are relatively easy to obtain in the factory to generate data for model training and evaluation in a reverse manner. The large language model 1022 is fine-tuned with this data, and prompt is used as guidance information. It is spliced ​​with the natural description text and input into the large language model 1022 to reduce the hallucination problem of the large language model 1022, thereby completing the training; finally, the trained large language model 1022 is used to decompose the total task in natural language form to generate a subtask sequence 1023.

[0039] The intelligent factory map modeling module 30 is the basis of this system. It receives the existing video data in the intelligent manufacturing environment, and performs real-time environmental map modeling of the intelligent manufacturing environment through image distortion correction, obstacle segmentation and multiple local image stitching to generate a global environmental map. At the same time, it provides the necessary global map and environmental information for the multi-robot path planning and obstacle avoidance module 20, helping the system to accurately locate the robot position to plan a safe, efficient and reliable task path, and update the global environmental map in real time according to the movement changes of the multi-robot path planning and obstacle avoidance module 20;

[0040] Preferably, the smart factory map modeling module 30 includes image distortion correction 301, obstacle segmentation 302 and multi-camera image stitching 303; the cloud intelligently receives video data of multiple cameras in the smart manufacturing environment in real time, and then extracts the video image of each sub-area from the video data, and then performs image distortion correction 301 on the video image of each sub-area to obtain the corrected image of each sub-area, and then performs obstacle segmentation 302 on the corrected image of each sub-area to obtain the area map of each sub-area, and finally performs multi-camera image stitching 303 on the area map of each sub-area to generate a complete global environment map.

[0041] Image distortion correction 301 is a distortion correction method that eliminates image distortion caused by the physical characteristics of the camera lens;

[0042] Obstacle segmentation 302 is a contour calibration method that accurately identifies and locates obstacles, robots, and drivable roads in the area map, reducing the complexity of image stitching;

[0043] Multi-camera image stitching 303 is an image stitching method that stitches area maps generated by multiple cameras to construct an accurate global environment map, providing a basis for global path planning.

[0044] Preferably, the process of the intelligent factory map modeling module 30 is as follows: Figure 4 As shown, specifically: first, the cloud intelligently receives video data from multiple cameras in the intelligent manufacturing environment in real time, then extracts video images 3011 of each sub-area from the video data, and then generates corrected images 3013 of each sub-area through a distortion correction algorithm 3012 for the video images 3011 of each sub-area, thereby reducing image distortion caused by sensor errors and mechanical assembly problems; then, the corrected images 3013 of each sub-area are calibrated with obstacles, robots and passable roads in the environment through a semantic segmentation algorithm 3021, thereby reducing the impact of visual differences on the image stitching module, and generating regional maps 3022 of each sub-area; finally, the regional maps 3022 of each sub-area are stitched into a complete global environment map 3032 using a feature matching algorithm 3031.

[0045] The multi-robot path planning and obstacle avoidance module 20 is the main working body of this system. It is used to generate the optimal motion path for multiple robots to work together, and optimize the path in real time according to the dynamic changes of the environment, thereby optimizing the decision-making ability of cloud-based intelligence.

[0046] The multi-robot path planning and obstacle avoidance module 20 receives the global environment map 3032 generated by the intelligent factory map modeling module 30 and the robot subtasks 1034 of the multi-robot task scheduling and wireless communication protocol module 10, generates an initial path for each robot, and detects and dynamically adjusts the robot's motion path in real time during task execution. That is, when the motion path changes dynamically (such as workers suddenly entering the task scene, the location of goods changes, and other robot paths change), the robot feeds back to the cloud intelligence and requests to re-plan the path. At the same time, before receiving the new path sent back by the cloud intelligence, the robot performs local path adjustments based on its own intelligence to achieve autonomous obstacle avoidance, avoid collisions with other robots and other safety hazards.

[0047] Preferably, the multi-robot path planning and obstacle avoidance module 20 includes cloud-based robot path planning 201, robot-based collision detection and obstacle avoidance 202 and cloud-based active learning 203; the cloud-based intelligence performs cloud-based intelligence-based robot path planning 201 for each robot according to the sub-task instructions 1034 generated by the multi-robot task scheduling and wireless communication protocol module 10 and the global environment map 3032 generated by the intelligent factory map modeling module 30, and sends the instruction sequence of the initial path to the corresponding robot. If the motion path changes dynamically, the robot performs robot-based collision detection and obstacle avoidance 202; when the task is completed, combined with the environmental information collected during the movement of the multi-robots, the cloud-based intelligence is optimized by comprehensively considering the collision situation and time overhead to achieve cloud-based active learning 203.

[0048] The cloud-based robot path planning 201 utilizes the powerful computing resources of the cloud-based intelligence to generate an initial path for each robot according to the task requirements and positional relationships based on the global environment map 3032 and each robot subtask 1034; when receiving a request to re-plan the path from the robot end, the cloud-based intelligence optimizes the path based on the robot's status information and the updated global map, re-plans the path for the multi-robot system, and sends the new path sequence information to the corresponding robot through the cloud-machine communication protocol;

[0049] Robot-based collision detection and obstacle avoidance 202 monitors the robot's motion path in real time and predicts the motion trend of the moving target in response to the dynamic obstacles that may appear on the robot's motion path, thereby performing collision analysis. The robot then decides whether to adjust the path and autonomously avoid obstacles based on the collision analysis results. If autonomous obstacle avoidance is required, local path planning is performed until the optimized path from the cloud is received again. The robot's local motion path is dynamically updated to avoid collisions and other potential dangerous situations, and the status information is fed back to the cloud intelligence.

[0050] Cloud-based active learning 203 aims to address the decision-making risks that may be caused by the lack of understanding of the task execution environment by cloud intelligence. An active learning method using a multi-robot system as the cloud intelligence sensor is designed to break the traditional supervised passive learning method of cloud intelligence, reduce dependence on labeled data, establish a comprehensive and three-dimensional perception of the environment by cloud intelligence, and realize cloud-based embodied intelligence.

[0051] Preferably, the process of the multi-robot path planning and obstacle avoidance module 20 is as follows: Figure 3 As shown, specifically: first, the cloud intelligence 2012 generates an initial path 2013 for the robot to execute according to the environmental state 2011 and sends it to each robot through the communication protocol 101; then the robot moves according to the initial path 2013 generated by the cloud intelligence, and identifies whether there is a moving target in the direction of movement according to its own video data. If there is a moving target, target recognition 2021 and trajectory prediction 2022 are performed; then the robot performs collision analysis 2023 on the moving target in the field of view according to the prediction results. If no collision occurs, the robot continues to move along the original path. If a collision occurs, the robot uses its own hardware equipment (such as camera, laser radar, etc.) ) obtains the local environmental status to perform autonomous obstacle avoidance 2024 and local path adjustment 2025 and feeds back the real-time status information to the cloud-based intelligence 2012; the cloud-based intelligence 2012 re-optimizes the path for the robot 2014 according to the changed robot status information and environmental status 2011, and sends the instruction sequence of the new path to the robot; finally, when all robots reach the target position, the decision-making performance of the cloud-based intelligence 2012 is evaluated by comprehensively considering the collision situation and time overhead, and the environmental information 2032 collected during the movement of the multi-robot system 2031 is combined to improve the decision-making ability of the cloud-based intelligence 2012 and enhance the embodied intelligence 2033 of the cloud.

[0052] The present invention also provides a collaborative method based on the multi-robot collaborative system (hereinafter referred to as the method), characterized in that the method comprises the following steps:

[0053] Step 1: The cloud-based intelligence receives the general task in the form of natural language sent by the manufacturing execution system, first decomposes the general task 102, sends the natural language instructions 1021 in the general task to the large language model 1022, and uses the semantic understanding ability and task planning ability of the large language model 1022 to generate a subtask sequence 1023 executable by the robot; then the cloud-based intelligence sends the task requirements 1031, the role functions 1032 and the position information of the robot as prior knowledge together with the subtask sequence 1023 to the expert large language model 1033, so that it generates each robot subtask 1034; finally, through the communication protocol 101 between the cloud-based intelligence and the heterogeneous multi-robots, each robot subtask 1034 is assigned to the corresponding robot;

[0054] The cloud intelligently receives video data from multiple cameras in the intelligent manufacturing environment in real time, extracts video images 3011 of each sub-area from the video data, and then performs image distortion correction 301. The video images 3011 of each sub-area are subjected to a distortion correction algorithm 3012 to obtain a corrected image 3013 of each sub-area; obstacle segmentation 302 is then performed. The corrected image 3021 of each sub-area is calibrated with obstacles, robots and drivable roads in the environment through a semantic segmentation algorithm 3022 to generate a regional map 3022 of each sub-area; finally, multi-camera image stitching 303 is performed, and the regional maps 3022 of each sub-area are stitched into a complete global environment map 3032 using a feature matching algorithm 3031;

[0055] Step 2: The cloud intelligence performs cloud intelligence-based robot path planning 201 for each robot according to the subtask instructions 1034 and the global environment map 3032, generates an initial path 2013 for task execution, and sends the instruction sequence of the initial path 2013 to the corresponding robot through the communication protocol 101 between the cloud and the robot;

[0056] Step 3, the robot receives and executes the command sequence from the cloud, and moves according to the initial path 2013 generated by the cloud intelligence; when the task environment changes dynamically, the robot performs robot-based collision detection and obstacle avoidance 202, and identifies whether there is a moving target in the direction of movement according to its own video data. If there is a moving target, it performs target recognition 2021 and trajectory prediction 2022; then the robot performs collision analysis 2023 on the moving target in the field of view according to the prediction results. If no collision occurs, it continues to move along the original path. If a collision occurs, it uses its own hardware equipment to obtain environmental information for autonomous obstacle avoidance 2024 and local path adjustment 2025, and feeds back real-time environmental information to the cloud intelligence 2012; the cloud intelligence 2012 re-optimizes the path for the robot according to the changed robot state information and environmental map information 2014, and sends the command sequence of the new path to the robot;

[0057] Step 4: When all robots reach the target position, the decision-making performance of the cloud intelligence 2012 is evaluated by comprehensively considering the collision situation and time cost, and the decision-making ability of the cloud intelligence 2012 is improved by combining the environmental information 2032 collected during the movement of the multi-robot system 2031, and the embodied intelligence 2033 of the cloud is enhanced to realize active learning 203 based on the cloud;

[0058] Step 5: After completing the above tasks, the multi-robot collaborative system waits for the manufacturing execution system to issue new tasks, and after receiving the new tasks, repeats steps 1-4 to enter the next round of task execution.

[0059] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A multi-robot collaborative system based on cloud-machine hybrid intelligent drive, characterized in that: The system comprises a multi-robot task scheduling and wireless communication protocol module (10), a multi-robot path planning and obstacle avoidance module (20) and an intelligent factory map modeling module (30); The multi-robot task scheduling and wireless communication protocol module (10) is used to decompose and distribute complex tasks and realize communication between cloud machines, and at the same time provides the multi-robot path planning and obstacle avoidance module (20) with information on the relationship between the subtasks and positions of each robot; when the task environment or task requirements change dynamically, feedback information from the multi-robot path planning and obstacle avoidance module (20) is received to replan the movement path of the robot; The intelligent factory map modeling module (30) receives the existing video data in the intelligent manufacturing environment, performs real-time environment map modeling of the intelligent manufacturing environment through image distortion correction, obstacle segmentation and multiple local image splicing, generates a global environment map, and provides the necessary global map and environment information for the multi-robot path planning and obstacle avoidance module (20), helps the system to accurately locate the robot position to plan the task path, and updates the global environment map in real time according to the movement changes of the multi-robot path planning and obstacle avoidance module (20); The multi-robot path planning and obstacle avoidance module (20) receives the global environment map (3032) generated by the intelligent factory map modeling module (30) and each robot subtask (1034) of the multi-robot task scheduling and wireless communication protocol module (10), generates an initial path for each robot, and detects and dynamically adjusts the robot's motion path in real time during task execution. That is, when the motion path changes dynamically, the robot feeds back to the cloud intelligence and requests to re-plan the path. At the same time, before receiving the new path sent back by the cloud intelligence, the robot adjusts the local path according to its own intelligence to achieve autonomous obstacle avoidance, avoid collision with other robots and other safety hazards.

2. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The multi-robot task scheduling and wireless communication protocol module (10) includes a communication protocol (101), task decomposition (102) and task allocation (103); after receiving the total task in natural language form sent by the existing manufacturing execution system of the factory, the cloud intelligence performs task decomposition (102) on the total task, and uses the communication protocol (101) between the cloud intelligence and the heterogeneous multi-robots to perform task allocation (103); The communication protocol (101) is a universal communication protocol between cloud intelligence and heterogeneous multi-robots, realizing real-time command interaction and feedback between robots and servers; Task decomposition (102) receives the general task in natural language form sent by the existing manufacturing execution system of the factory, and uses the semantic understanding ability and task planning ability of the large language model (1022) to decompose the complex task into several executable subtasks and generate a subtask sequence (1023); The task allocation (103) trains the existing large language model (1022), comprehensively considers the nature of the subtasks, the functions of different robots and environmental information, and allocates the generated robot subtasks (1034) to the corresponding robots through the communication protocol (101).

3. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The process of the multi-robot task scheduling and wireless communication protocol module (10) is as follows: first, the cloud-based intelligence receives the general task in the form of natural language sent by the manufacturing execution system, sends the natural language instructions (1021) in the general task to the large language model (1022), and uses the semantic understanding ability and task planning ability of the large language model (1022) to generate a robot-executable subtask sequence (1023); then the cloud-based intelligence sends the task requirements (1031), the robot's role function (1032) and position information as prior knowledge combined with the subtask sequence (1023) to the expert large language model (1033), so that it generates each robot subtask (1034); finally, each robot subtask (1034) is packaged according to the requirements of the message structure (1011), the message flow (1012) and the message transmission mechanism (1013) and sent to the corresponding robot.

4. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 3 is characterized in that: The semantic understanding and task planning capabilities of the large language model (1022) are used to generate a robot-executable subtask sequence (1023). Specifically, the cloud-based intelligence constructs a task decomposition dataset in a bottom-up manner, and uses subtasks that are relatively easy to obtain in the factory to generate data for model training and evaluation in a reverse manner. The large language model (1022) is fine-tuned using this data. At the same time, the prompt is used as guidance information, which is spliced ​​with the natural description text and then input into the large language model (1022) to reduce the hallucination problem of the large language model (1022), thereby completing the training. Finally, the trained large language model (1022) is used to decompose the total task in natural language form to generate a subtask sequence (1023).

5. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The intelligent factory map modeling module (30) includes image distortion correction (301), obstacle segmentation (302) and multi-camera image stitching (303); the cloud intelligently receives video data of multiple cameras in the intelligent manufacturing environment in real time, extracts video images of each sub-area from the video data, performs image distortion correction (301) on the video images of each sub-area to obtain a corrected image of each sub-area, performs obstacle segmentation (302) on the corrected image of each sub-area to obtain a regional map of each sub-area, and finally performs multi-camera image stitching (303) on the regional map of each sub-area to generate a complete global environment map; Image distortion correction (301) is a distortion correction method that eliminates image distortion caused by the physical characteristics of the camera lens; Obstacle segmentation (302) is a contour calibration method that accurately identifies and locates obstacles, robots and drivable roads in the regional map, reducing the complexity of image stitching; Multi-camera image stitching (303) is an image stitching method that stitches area maps generated by multiple cameras to construct an accurate global environment map, providing a basis for global path planning.

6. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The process of the intelligent factory map modeling module (30) is as follows: first, the cloud intelligently receives video data from multiple cameras in the intelligent manufacturing environment in real time, then extracts the video image (3011) of each sub-area from the video data, and then generates the corrected image (3013) of each sub-area through the distortion correction algorithm (3012) of the video image (3011) of each sub-area; then, the corrected image (3013) of each sub-area is calibrated with obstacles, robots and drivable roads in the environment through the semantic segmentation algorithm (3021), and the area map (3022) of each sub-area is generated; finally, the feature matching algorithm (3031) is used to splice the area maps (3022) of each sub-area into a complete global environment map (3032).

7. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The multi-robot path planning and obstacle avoidance module (20) includes cloud-based robot path planning (201), robot-based collision detection and obstacle avoidance (202) and cloud-based active learning (203); the cloud-based intelligence performs cloud-based robot path planning (201) for each robot based on the subtask instructions (1034) generated by the multi-robot task scheduling and wireless communication protocol module (10) and the global environment map (3032) generated by the intelligent factory map modeling module (30), and sends the instruction sequence of the initial path to the corresponding robot. If the motion path changes dynamically, the robot performs robot-based collision detection and obstacle avoidance (202); when the task is completed, the cloud-based intelligence is optimized by comprehensively considering the collision situation and time overhead in combination with the environmental information collected during the multi-robot movement, so as to realize cloud-based active learning (203); The cloud-based robot path planning (201) utilizes the powerful computing resources of the cloud-based intelligence, based on the global environment map (3032) and each robot subtask (1034), to generate an initial path for each robot according to the task requirements and positional relationships; when receiving a request for re-planning the path from the robot side, the cloud-based intelligence optimizes the path based on the robot's status information and the updated global map, re-plans the path for the multi-robot system, and sends the new path sequence information to the corresponding robot through the cloud-machine communication protocol; The robot-based collision detection and obstacle avoidance (202) monitors the robot's motion path in real time and predicts the motion trend of the moving target in response to the dynamic obstacles that may appear on the robot's motion path, thereby performing collision analysis. The robot then decides whether to adjust the path and autonomously avoid obstacles based on the collision analysis results. If autonomous obstacle avoidance is required, local path planning is performed until the optimized path from the cloud is received again. The robot's local motion path is dynamically updated to avoid collisions and other potential dangerous situations, and the status information is fed back to the cloud intelligence. Cloud-based active learning (203) designs an active learning method using a multi-robot system as a cloud-based intelligent sensory organ, establishes a comprehensive three-dimensional perception of the environment by cloud-based intelligence, and realizes cloud-based embodied intelligence.

8. The multi-robot collaborative system based on cloud-machine hybrid intelligent drive according to claim 1 is characterized in that: The process of the multi-robot path planning and obstacle avoidance module (20) is as follows: first, the cloud intelligence (2012) generates an initial path (2013) for the robot to execute according to the environmental state (2011) and sends it to each robot through the communication protocol (101); then the robot moves according to the initial path (2013) generated by the cloud intelligence, and identifies whether there is a moving target in the direction of movement according to its own video data. If there is a moving target, target recognition (2021) and trajectory prediction (2022) are performed; then the robot performs collision analysis (2023) on the moving target in the field of view according to the prediction result. If no collision occurs, the robot continues to move along the original path. If a collision occurs, the robot uses its own hardware device to obtain The local environmental status is obtained to perform autonomous obstacle avoidance (2024) and local path adjustment (2025), and the real-time status information is fed back to the cloud intelligence (2012); the cloud intelligence (2012) re-optimizes the path for the robot according to the changed robot status information and environmental status (2011) (2014), and sends the instruction sequence of the new path to the robot; finally, when all robots reach the target position, the decision-making performance of the cloud intelligence (2012) is evaluated by comprehensively considering the collision situation and time cost, and the decision-making ability of the cloud intelligence (2012) is improved by combining the environmental information (232) collected during the movement of the multi-robot system (2031), thereby enhancing the embodied intelligence of the cloud (2033).

Citation Information

Cited By

  • Heterogeneous robot cluster management platform and management method thereof

    CN119996462A

  • Three-dimensional modeling design method and system based on multi-machine cooperation of building robots

    CN120297628A

  • Three-dimensional modeling design method and system based on multi-machine cooperation of construction robots

    CN120297628B

  • Multi-robot autonomous cooperation method and system

    CN120773049A

  • Intelligent robot control system based on Internet of Things

    CN120862704A