Heterogeneous robot collaborative work system, method, device and medium

Abstract

The application provides an isomeric robot cooperative work system, method, device and medium, relates to the technical field of robots, and the system comprises: a task planning unit splits a work task into multiple work subtasks; a task execution unit sends a cooperative work request to a dynamic viewpoint planning unit when it is determined that there is an operation blind area according to environment perception data; the dynamic viewpoint planning unit acquires a three-dimensional semantic map from a fusion map construction unit when the cooperative work request is received; a cooperative control instruction is generated based on the operation blind area and the three-dimensional semantic map and is sent to a slave work robot through a first cooperative communication module; the slave work robot moves to a target position and adjusts to a target attitude according to the cooperative control instruction, acquires a regional image of the operation blind area and sends the regional image to the fusion map construction unit; and the fusion map construction unit fuses the regional image and the three-dimensional semantic map to obtain a panoramic fusion map. The application realizes deep perception fusion and physical cooperative work of master-slave work robots.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and more specifically, to a heterogeneous robot collaborative operation system, method, device, and medium. Background Technology

[0002] With the rapid development of robotics technology, robots are increasingly being used in complex environments, such as disaster relief, industrial equipment inspection, and pipeline system maintenance. These environments are typically characterized by complex spatial structures and diverse operational requirements. Due to limitations in form and function, a single type of robot cannot independently complete all tasks. Therefore, how to effectively coordinate heterogeneous robots of different forms and functions to leverage their respective advantages has become an important research direction in the field of robotics.

[0003] While existing multi-robot collaborative operation systems have expanded operational capabilities to some extent, their collaborative mechanisms have significant shortcomings—they lack deep perception fusion capabilities and struggle to achieve dynamic collaboration at the physical level. Summary of the Invention

[0004] In view of this, the purpose of this application is to overcome the shortcomings of the prior art and provide a heterogeneous robot collaborative operation system, method, device, and medium. This application provides the following technical solution:

[0005] In a first aspect, the present invention provides a heterogeneous robot collaborative operation system, the system comprising: a master robot and slave robots;

[0006] The main operating robot includes a main control module and a first collaborative communication module. The main control module includes a task planning unit, a task execution unit, a dynamic viewpoint planning unit, and a fusion map construction unit.

[0007] The task planning unit is used to obtain the user-input task and divide the task into N ordered sub-tasks according to a preset task decomposition strategy.

[0008] The task execution unit is used to acquire environmental perception data during the execution of the i-th subtask and determine whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to the dynamic viewpoint planning unit.

[0009] The dynamic viewpoint planning unit is used to obtain a three-dimensional semantic map of the current environment from the fusion map construction unit when the collaborative operation request is received; and to generate collaborative control instructions based on the spatial location and geometric features of the operation blind spot in the three-dimensional semantic map, combined with the motion capability constraints of the operation robot and the environmental safety conditions.

[0010] The first collaborative communication module is used to send the collaborative control command to the slave robot;

[0011] The robot is used to move to the target location and adjust itself to the target posture according to the cooperative control command, so as to obtain an area image of the operation blind zone; and send the area image to the fusion map construction unit.

[0012] The fusion map construction unit is used to perform panoramic fusion of the regional image and the three-dimensional semantic map to obtain a panoramic fusion map.

[0013] In one embodiment, the main control module further includes: a physical coordination management unit;

[0014] The task execution unit is also used to obtain the operation accuracy during the execution of the i-th job sub-task. If the operation accuracy is lower than a preset accuracy threshold, a support job request is sent to the physical collaboration management unit.

[0015] The physical collaboration management unit is used to generate support control commands based on the operating posture and expected force direction of the main robot included in the i-th sub-task when it receives the support operation request.

[0016] The first collaborative communication module is also used to send the support control command to the slave robot;

[0017] The slave robot is used to move to the target support position according to the support control command, and after adjusting itself to the target support posture, it enters the posture locking mode to provide physical support for the master robot to perform the i-th sub-task.

[0018] In one embodiment, the main operating robot further includes an interaction module, which includes a rendering unit;

[0019] The rendering unit is used to obtain the panoramic fusion map from the fusion map construction unit, and perform image rendering based on the panoramic fusion map to obtain a visual rendering image representing the current environment.

[0020] In one embodiment, the robot includes: a second collaborative communication module and a motion control module;

[0021] The second collaborative communication module is used to obtain the collaborative control command from the first collaborative communication module, parse the collaborative control command to obtain multiple control parameters, and send motion control signals to the motion control module according to each of the control parameters;

[0022] The motion control module is used to control the robot body to move to the target position and adjust itself to the target posture according to the control signal.

[0023] In one embodiment, the robot further includes an image acquisition module;

[0024] The image acquisition module is used to acquire the region image of the operation blind zone and send the region image to the second cooperative communication module;

[0025] The second collaborative communication module is also used to send the regional image to the first collaborative communication module.

[0026] In one embodiment, the step of performing panoramic fusion of the region image and the three-dimensional semantic map to obtain a panoramic fused map includes:

[0027] Obtain the pre-calibrated fixed transformation matrix between the image acquisition module and the interaction module;

[0028] Obtain the real-time poses of the master robot and the slave robot;

[0029] Based on the fixed transformation matrix and the real-time pose, a coordinate transformation chain is constructed;

[0030] Based on the coordinate transformation chain, calculate the projection relationship of the region image in the three-dimensional semantic map coordinate system;

[0031] Based on the projection relationship, the regional image is overlaid and fused with the three-dimensional semantic map to obtain the panoramic fused map.

[0032] In one embodiment, before overlaying and fusing the region image with the three-dimensional semantic map according to the projection relationship, the method further includes:

[0033] The image of the region is subjected to time synchronization processing.

[0034] Secondly, the present invention provides a method for heterogeneous robot collaborative operation, applied to a heterogeneous robot collaborative operation system as described in any of the foregoing embodiments, the method comprising:

[0035] The task planning unit obtains the user-input task and breaks it down into N ordered sub-tasks according to a preset task decomposition strategy.

[0036] The task execution unit obtains environmental perception data during the execution of the i-th job subtask and determines whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to the dynamic viewpoint planning unit.

[0037] When the collaborative operation request is received, the dynamic viewpoint planning unit obtains a three-dimensional semantic map of the current environment from the fusion map construction unit; based on the spatial location and geometric features of the operation blind spot in the three-dimensional semantic map, and combined with the motion capability constraints of the operation robot and the environmental safety conditions, a collaborative control command is generated.

[0038] The first collaborative communication module sends the collaborative control command to the slave robot.

[0039] The robot moves to the target location according to the cooperative control instructions and adjusts itself to the target posture to obtain an area image of the operation blind zone; the area image is then sent to the fusion map construction unit.

[0040] The fusion map construction unit performs panoramic fusion of the regional image and the three-dimensional semantic map to obtain a panoramic fusion map.

[0041] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program executes the heterogeneous robot collaborative operation method described in the foregoing embodiments when it is run on the processor.

[0042] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the heterogeneous robot collaborative operation method described in the foregoing embodiments.

[0043] The beneficial effects of this invention are: it realizes the deep perception fusion and physical collaborative operation between the master robot and the slave robot, and extends the perception range and improves the operation accuracy of the heterogeneous robot collaborative operation system.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A schematic diagram of a heterogeneous robot collaborative operation system provided in an embodiment of this application is shown;

[0047] Figure 2Another structural schematic diagram of the heterogeneous robot collaborative operation system provided in this application embodiment is shown;

[0048] Figure 3 This paper shows another structural schematic diagram of the heterogeneous robot collaborative operation system provided in an embodiment of this application;

[0049] Figure 4 This paper presents another structural schematic diagram of the heterogeneous robot collaborative operation system provided in an embodiment of this application;

[0050] Figure 5 A flowchart illustrating a heterogeneous robot collaborative operation method provided in an embodiment of this application is shown.

[0051] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0052] Explanation of key component symbols:

[0053] 1000 - Heterogeneous robot collaborative operation system; 1100 - Main robot; 1110 - Main control module; 1111 - Task planning unit; 1112 - Task execution unit; 1113 - Dynamic viewpoint planning unit; 1114 - Fusion map construction unit; 1115 - Physical collaborative management unit; 1120 - First collaborative communication module; 1130 - Interaction module; 1200 - Slave robot; 1210 - Second collaborative communication module; 1220 - Motion control module; 1230 - Image acquisition module; 600 - Electronic device; 601 - Transceiver; 602 - Processor; 603 - Memory. Detailed Implementation

[0054] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0057] Example 1

[0058] Single-type robots are insufficient to meet the operational needs of complex environments. How heterogeneous robots of different forms can work collaboratively has become an important research direction in the field of robotics. Existing heterogeneous robot collaborative systems lack deep perception fusion capabilities, exhibit superficial collaboration, and only reach the level of simple state data sharing. They fail to integrate the perception channels of heterogeneous robots and struggle to achieve dynamic collaboration at the physical level. For further information, please refer to [link to relevant documentation / reference]. Figure 1 This application provides a heterogeneous robot collaborative operation system 1000, including: a master robot 1100 and a slave robot 1200;

[0059] The main operation robot 1100 includes: a main control module 1110 and a first collaborative communication module 1120. The main control module 1110 includes: a task planning unit 1111, a task execution unit 1112, a dynamic viewpoint planning unit 1113, and a fusion map construction unit 1114.

[0060] The task planning unit 1111 is used to obtain the user-inputted task and divide the task into N ordered sub-tasks according to a preset task decomposition strategy.

[0061] The task execution unit 1112 is used to acquire environmental perception data during the execution of the i-th subtask and determine whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to the dynamic viewpoint planning unit 1113.

[0062] The dynamic viewpoint planning unit 1113 is used to obtain a three-dimensional semantic map of the current environment from the fusion map construction unit 1114 when the collaborative operation request is received; and generate collaborative control instructions based on the spatial position and geometric features of the operation blind spot in the three-dimensional semantic map, combined with the motion capability constraints of the operation robot 1200 and the environmental safety conditions.

[0063] The first collaborative communication module 1120 is used to send the collaborative control command to the slave robot 1200;

[0064] The robot 1200 is used to move to the target position and adjust itself to the target posture according to the cooperative control command, so as to obtain the area image of the operation blind zone; and send the area image to the fusion map construction unit 1114.

[0065] The fusion map construction unit 1114 is used to perform panoramic fusion of the regional image and the three-dimensional semantic map to obtain a panoramic fusion map.

[0066] In this embodiment, the task planning unit 1111 breaks down the complex task input by the user into multiple ordered and executable sub-tasks. For example, the task "inspect and repair equipment faults" is broken down into the following ordered sub-tasks: the main operation robot 1100 moves to the designated position on the front of the equipment; the operation robot 1200 explores the blind spot at the bottom of the equipment; and the main operation robot 1100 performs fault repair based on the exploration results.

[0067] During the execution of the sub-task, the task execution unit 1112 will collect environmental perception data in real time. For example, it will capture images of the equipment to be repaired by the camera of the main robot 1100 and scan the surrounding environmental data by LiDAR. The task execution unit 1112 will analyze the environmental perception data to determine whether there are blind spots. For example, if the bottom of the equipment is blocked, the main robot 1100 will not be able to capture the image of the bottom of the equipment and will determine that there are blind spots. Then, it will send a collaborative operation request to the dynamic viewpoint planning unit 1113.

[0068] After receiving a collaborative operation request, the dynamic viewpoint planning unit 1113 first obtains a three-dimensional semantic map of the current environment from the fusion map construction unit 1114 (the map includes semantic labels and three-dimensional coordinates of the equipment to be repaired, the ground, obstacles, etc.); then, combined with the spatial location of the operation blind spot in the three-dimensional semantic map, such as the bottom coordinates of the equipment to be repaired (X=10m, Y=5m, Z=0.3m), geometric features such as narrow space and height ≤0.4m, and the motion capability constraints of the working robot 1200, such as the ability to crawl through spaces with a height ≥0.2m and environmental safety conditions such as no sharp obstacles on the ground and no risk of leakage, generates a collaborative control command and sends it to the working robot 1200 through the first collaborative communication module 1120 to control the working robot 1200 to perform collaborative operations. Specifically, the dynamic viewpoint planning unit 1113 determines the 3D bounding box of the operation blind zone based on the 3D semantic map. Then, based on the viewpoint quality evaluation function, it samples and calculates a set of optimal observation points around the 3D bounding box and checks whether these observation points are within the motion capability and environmental safety conditions of the working robot (e.g., will not cause the working robot to get stuck or fall). If so, a smooth gimbal motion trajectory is generated based on these observation points. The cooperative control command is used to control the working robot to perform cooperative operations according to this gimbal motion trajectory.

[0069] It should be noted that the first collaborative communication module 1120 uses a custom application layer protocol based on User Datagram Protocol (UDP) for data transmission. While ensuring low latency, it implements forward error correction and packet loss retransmission mechanisms to cope with unstable field network environments.

[0070] After receiving collaborative control instructions from the work robot 1200, such as a robot dog, it moves itself to the target position and adjusts its body posture to the target posture, such as lying down to lower its height to 0.3m, and controls the gimbal camera to aim at the blind spot, turns on the lights to obtain clear area images, such as high-definition images of cracks in bottom parts, and then sends the area image back to the fusion map building unit 1114 of the main work robot 1100.

[0071] After receiving the blind zone image transmitted from the working robot 1200, the fusion map construction unit 1114 uses the pre-calibrated camera parameters and the real-time pose of the master and slave robots to construct a coordinate transformation chain, accurately overlaying the area image onto the original three-dimensional semantic map. For example, the image of the bottom crack is pasted onto the corresponding position of the device to be repaired in the semantic map, and finally a panoramic fusion map containing blind zone information is formed, allowing the master working robot 1100 to fully grasp the overall environment.

[0072] Its beneficial effects are as follows: It enables orderly control of the work process through task decomposition, avoiding chaotic task execution; it effectively solves the problems of fixed robot perception range and blind spots by using environmental perception data to identify blind spots in real time and trigger collaboration; it ensures the accuracy and safety of the operation of the 1200 robot by generating instructions based on the 3D semantic map and the motion and safety constraints of the 1200 robot; it forms a unified and complete perception space by collecting blind spot images from the 1200 robot and merging them with the 3D semantic map, eliminating the need for manual piecing together of multi-source separated perspective information and significantly extending the overall perception range; the panoramic fusion map provides comprehensive and accurate environmental basis for subsequent operation decisions, improving the efficiency and reliability of heterogeneous robot collaborative operations and overcoming the limitations of insufficient perception fusion in existing multi-robot collaboration.

[0073] In one implementation, please refer to Figure 2 The main control module 1110 further includes: a physical coordination management unit 1115;

[0074] The task execution unit 1112 is also used to obtain the operation accuracy during the execution of the i-th job sub-task. If the operation accuracy is lower than a preset accuracy threshold, a support job request is sent to the physical collaboration management unit 1115.

[0075] The physical collaboration management unit 1115 is used to generate support control commands based on the operating posture and expected force direction of the main robot 1100 included in the i-th sub-task when the support operation request is received.

[0076] The first collaborative communication module 1120 is also used to send the support control command to the slave robot 1200;

[0077] The slave robot 1200 is used to move to the target support position according to the support control command, and after adjusting itself to the target support posture, enter the posture locking mode to provide physical support for the master robot 1100 to execute the i-th sub-task.

[0078] In this embodiment, when the main robot 1100, specifically a humanoid robot, is performing the i-th sub-task (such as circuit board solder joint repair), the task execution unit 1112 collects operational accuracy data in real time, such as the position deviation of the end effector and the amplitude of motion jitter, and compares it with a preset accuracy threshold. If the operational accuracy is detected to be lower than the preset accuracy threshold, it is determined that the current task requires external stable support, and then a support task request is sent to the physical collaboration management unit 1115 of the main control module 1110 to start the physical collaboration process.

[0079] After receiving a support operation request, the physical collaborative management unit 1115 retrieves key parameters of the i-th subtask, such as the operating posture of the main robot 1100 (e.g., the extension angle of the robotic arm, the three-dimensional coordinates of the end effector above the circuit board X=0.8m, Y=0.5m, Z=0.1m) and the expected force direction (e.g., the downward pressure direction during welding, the lateral force direction during horizontal fine-tuning). Combining this with the motion capabilities and stable support requirements of the secondary robot 1200, it generates precise support control commands. These commands include: the target support position (X=0.8m, Y=0.5m, Z=0.05m), the target support posture (body height 0.3m, limbs extended in a stable triangular support, gimbal facing the operating direction of the main robot 1100), and posture lock trigger conditions. The core algorithm of this process is workspace intersection analysis: calculating the spatial intersection between the reachable workspace of the secondary robot and the required support area of ​​the main robot, thereby finding the spatial area that the secondary robot can safely reach and effectively support the operation of the main robot. The specific process includes:

[0080] Define the required support area for the main robot: During fine manipulation, not only does the end point of the main robot need support, but the entire manipulator arm may also require a reaction force to suppress vibration during movement. Therefore, based on the operating pose and expected force direction of the main robot, it is modeled as a polygonal prism or ellipsoid centered on the target point P_tool of the main robot's operating end point and extending along the possible movement direction of the manipulator arm.

[0081] For example, if a humanoid robot is to perform welding, its "required support area" might be a cylindrical space 10 centimeters high and 15 centimeters in radius above the welding path.

[0082] Calculate the reachable workspace of the robot: The workspace of the robot is the set of all points that its feet can reach through leg movements. Since the robot's body itself also has dimensions, the workspace of its effective support points is a subset of the foot workspace.

[0083] Calculation method: Numerical sampling method is used: Tens of thousands of possible foot position combinations are randomly generated from the robot's base coordinate system. Inverse kinematics is used to determine whether the posture is reachable and stable (no collision with the environment, joint angles within limits). The set of all feasible body poses constitutes its reachable workspace W_dog.

[0084] Computational spatial intersection: The "required support area" R_robot of the humanoid robot is transformed into the same global coordinate system as the workspace W_dog of the working robot through coordinate transformation. The intersection I = R_robot ∩ W_dog is the theoretically feasible set of support points.

[0085] Stability optimization: Within the intersection region, with the objective of maximizing the overall system stability margin (usually achieved by optimizing the ZMP stable region), the optimal support pose {x, y, z, yaw} for the robot is solved. This pose must ensure that the joint system does not become unstable when the humanoid robot applies forces.

[0086] Solving for the optimal support pose is a constrained optimization problem, and its mathematical expression is:

[0087] X=argmax x [Stability_Margin(X)] × subject to X ∈ I. Where: X = {x, y, z, yaw} represents the pose (position and yaw angle) of the robot; I is the intersection of the workspaces, serving as a constraint; Stability_Margin(X) is the objective function, calculating the stability margin under pose X. The stability margin calculation is based on the Zero Moment Point (ZMP) theory, specifically including the following key formulas:

[0088] a) Calculation of the centroid of the joint system: COMtotal = ,in: , These represent the masses of the master robot and the slave robot, respectively. , These represent the positions of the centroids of the two entities respectively; COMtotal represents the total centroid of the two entities.

[0089] b) Zero Moment Point (ZMP) Calculation: ZMP_x = x_c - ·a_x;ZMP_y = y_c - ·a_y. Where ZMP_x, ZMP_y represent the coordinates of the zero-torque point on the horizontal ground; x_c, y_c, z_c represent the coordinates of the center of mass of the combined system; a_x, a_y represent the horizontal acceleration of the center of mass; g represents the gravitational acceleration;

[0090] c) Stability margin calculation: S(X) = min(distance(ZMP, boundary_i)). Where boundary_i represents the i-th edge of the supporting polygon; min(distance(ZMP, boundary_i)) calculates the shortest distance from ZMP to each edge.

[0091] The first collaborative communication module 1120 sends the support control commands generated by the physical collaborative management unit 1115 to the slave robot 1200 through a low-latency communication channel, ensuring that the command transmission delay is controlled within a preset range and avoiding the impact of command lag on collaborative accuracy.

[0092] After receiving support control commands from the main robot 1200, the robot plans the optimal path through its own motion system and moves to the target support position, such as the stable area under the circuit board. Then, it adjusts its posture and moves its center of gravity to a stable point within the support polygon by fine-tuning its leg joints to ensure that it will not tip over when subjected to force. Finally, it enters the posture locking mode and switches all joint motors to high-gain impedance control mode to increase the virtual stiffness of the joints and make the body form a quasi-rigid structure. This provides a stable physical support surface for the robotic arm of the main robot 1100, enabling the main robot 1100 to rely on the stable support provided by the main robot 1200 to suppress the accuracy impact caused by its own standing posture micro-movements and joint servo jitter, and accurately execute the work sub-tasks.

[0093] Its beneficial effects are that, through the physical collaboration between the main robot 1100 and the slave robot 1200, the operation accuracy and stability of the main robot 1100 are greatly improved, avoiding operation failure or workpiece damage caused by the main robot 1100's own insufficient stability. At the same time, it can complete the support response without human intervention, reducing the operation risk and labor cost in complex scenarios.

[0094] In one implementation, please refer to Figure 3 The main working robot 1100 further includes an interaction module 1130, which includes a rendering unit. The rendering unit is used to obtain the panoramic fusion map from the fusion map construction unit 1114 and perform image rendering based on the panoramic fusion map to obtain a visual rendering image representing the current environment.

[0095] In this embodiment, the interaction module 1130 of the main working robot 1100 includes a rendering unit. The rendering unit is used to convert the panoramic fusion map generated by the fusion map construction unit 1114 into an intuitive and interpretable visual rendering image, and present it on the AR display screen of the main working robot 1100. The specific presentation method can be either a panoramic fusion mode: pasting the blind spot image onto the corresponding position on the three-dimensional semantic map to form a complete 3D environment picture; or a picture-in-picture mode: displaying the real-time observation picture of the slave working robot 1200 on the side of the main view screen, and marking the communication status and pose information of the slave working robot 1200 with a border, so that the user can intuitively grasp the overall picture of the environment and key information.

[0096] Its beneficial effects are that it effectively solves the drawbacks of existing multi-robot collaboration that require manual piecing together of separate perspectives and parsing of abstract data, significantly improves the efficiency and accuracy of operational decisions, provides clear and intuitive interactive support for collaborative operations in complex environments, and further expands the adaptability of heterogeneous robot systems in high-precision and complex scenarios.

[0097] In one implementation, please refer to Figure 4 The working robot 1200 includes: a second collaborative communication module 1210 and a motion control module 1220;

[0098] The second collaborative communication module 1210 is used to obtain the collaborative control instruction from the first collaborative communication module 1120, parse the collaborative control instruction to obtain multiple control parameters, and send motion control signals to the motion control module 1220 according to each of the control parameters.

[0099] The motion control module 1220 is used to control the robot body 1200 to move to the target position and adjust itself to the target posture according to the control signal.

[0100] In this embodiment, the second collaborative communication module 1210 of the work robot 1200 serves as the command receiving and parsing hub. It first obtains collaborative control commands, such as commands to probe the blind spot at the bottom of the equipment to be repaired, from the first collaborative communication module 1120 of the main work robot 1100 via a low-latency protocol matched with that of the first collaborative communication module 1120. The collaborative control commands are parsed to extract multiple key control parameters, including target position coordinates, gimbal pitch angle, movement speed threshold, and attitude adjustment time. These control parameters are then converted into standardized motion control signals recognizable by the motion control module 1220 and sent to it. Upon receiving the motion control signal, the motion control module 1220 uses its internal proportional-integral-derivative (PID) control algorithm and path planning logic to control the work robot 1200 to move along the planned path, accurately reaching the target position. Simultaneously, it controls the movable gimbal camera module to adjust its attitude, ensuring the gimbal camera is aligned with the blind spot direction, thus meeting the observation requirements of the main work robot 1100.

[0101] Its beneficial effects are that it achieves low-latency transmission and precise landing of collaborative control commands, avoiding command loss or delay in complex environments. Furthermore, by accurately analyzing control parameters and driving the precise movement of the slave robot 1200, it ensures that the slave robot 1200 can quickly respond to the collaborative needs of the master robot 1100, accurately reach the target position, and adjust to the optimal posture. This effectively solves the problems of rigid command interaction and large deviations in action execution in existing multi-robot collaborations, and provides reliable execution guarantees for deep perception fusion (such as blind spot image acquisition) and real-time physical collaboration (such as stable support), significantly improving the accuracy and efficiency of heterogeneous robot collaborative operations.

[0102] In one embodiment, the robot 1200 further includes an image acquisition module 1230;

[0103] The image acquisition module 1230 is used to acquire the region image of the operation blind zone and send the region image to the second cooperative communication module 1210.

[0104] The second collaborative communication module 1210 is also used to send the region image to the first collaborative communication module 1120.

[0105] In this embodiment, the image acquisition module 1230 of the working robot 1200 is used to collect visual information of the operation blind spot. It integrates an RGB camera, a depth camera, and a high-brightness LED light, and has a shock absorption and temperature control structure to ensure the stability of the acquisition in complex environments. For example, in a large pressure vessel inspection task, after the main working robot 1100 discovers that the inner side of the bottom of the container is an operation blind spot and sends a collaborative control command, the working robot 1200 moves to the target position at the bottom of the container and adjusts to a suitable posture. The image acquisition module 1230 then turns on the LED light to supplement the illumination, captures the operation blind spot image through the RGB camera, and simultaneously collects three-dimensional distance information through the depth camera to form a region image containing texture and depth. Subsequently, the image acquisition module 1230 performs local hardware encoding on the original region image, adds a precise timestamp and its own pose label, generates a standardized data packet, and sends it to the second collaborative communication module 1210 of the working robot 1200. After receiving the data packet, the second collaborative communication module 1210 transmits the regional image data packet through a low-latency UDP protocol stack matched with the first collaborative communication module 1120 of the main robot 1100 via a medium-priority channel. At the same time, it dynamically adapts to network quality based on the link health management mechanism to ensure that the data packet is stably sent to the first collaborative communication module 1120 of the main robot 1100.

[0106] Its beneficial effects are that it enables the accurate acquisition and reliable transmission of visual information in the blind spot of operation, allowing the main robot 1100 to seamlessly acquire blind spot information and form a unified perception space, significantly extending the overall perception range and improving the continuity and reliability of heterogeneous robot collaborative operation.

[0107] In one embodiment, the step of performing panoramic fusion of the region image and the three-dimensional semantic map to obtain a panoramic fused map includes:

[0108] Obtain the pre-calibrated fixed transformation matrix between the image acquisition module 1230 and the interaction module 1130;

[0109] Obtain the real-time poses of the master robot 1100 and the slave robot 1200;

[0110] Based on the fixed transformation matrix and the real-time pose, a coordinate transformation chain is constructed;

[0111] Based on the coordinate transformation chain, calculate the projection relationship of the region image in the three-dimensional semantic map coordinate system;

[0112] Based on the projection relationship, the regional image is overlaid and fused with the three-dimensional semantic map to obtain the panoramic fused map.

[0113] In this embodiment, the fixed transformation matrix is ​​a key parameter obtained in advance through joint calibration. It is used to clarify the fixed spatial relationship between the image acquisition module 1230 of the working robot 1200 and the interaction module 1130 of the main working robot 1100, and includes data such as extrinsic parameter matrix and camera intrinsic parameters.

[0114] The real-time global poses of the master robot 1100 and the slave robot 1200 are acquired to ensure the capture of their dynamic position changes during operation. The aforementioned fixed transformation matrix is ​​integrated with the real-time pose data of the master and slave robots to construct a complete coordinate transformation chain. This involves establishing a transformation path from the local coordinate system of the image acquisition module 1230 to the global coordinate system of the 3D semantic map through matrix operations, resolving the issue of inconsistent coordinate systems across different devices. For example, the coordinates of images captured by the slave robot 1200 are mapped to the coordinate system of the global 3D semantic map through the transformation chain.

[0115] Based on the constructed coordinate transformation chain, the corresponding position of each pixel in the regional image in the 3D semantic map coordinate system is accurately calculated, forming a one-to-one correspondence between "pixel point - 3D coordinate". According to the calculated projection relationship, the texture information of the regional image is superimposed and fused with the geometric structure and semantic labels of the 3D semantic map. This preserves the spatial structure and semantic attributes of the 3D semantic map while supplementing the detailed visual information of the regional image, ultimately forming a panoramic fusion map.

[0116] Its beneficial effect is that it effectively solves the problems of isolated perception data and separated perspectives in existing multi-robot collaboration, forming a unified and complete global perception space, allowing the main robot 1100 to grasp the overall picture of the environment and the details of blind spots without manually piecing together multi-source data.

[0117] In one embodiment, before overlaying and fusing the region image with the three-dimensional semantic map according to the projection relationship, the method further includes: performing time synchronization processing on the region image.

[0118] It is understandable that during the process of transmitting the regional image from the working robot 1200 to the fusion map building unit 1114 of the main working robot 1100, there are acquisition delays, encoding delays, wireless transmission delays and data processing delays, which cause a deviation between the actual image generation time and the main robot's reception time.

[0119] In this embodiment, the total delay is calculated based on the real-time pose synchronization time of the master robot 1100 and the slave robot 1200. The timestamp of the region image is then calibrated to the current real-time update timestamp of the 3D semantic map to eliminate the time difference between the region image and the 3D semantic map caused by delays in multiple stages.

[0120] Its beneficial effect is that it avoids spatial misalignment when regional images and 3D semantic maps are overlaid and fused due to time asynchrony, thus ensuring the spatial accuracy and data reliability of the panoramic fused map.

[0121] This application provides a heterogeneous robot collaborative operation system 1000, which includes a master robot 1100 and slave robots 1200. The master robot 1100 includes a master control module 1110 and a first collaborative communication module 1120. The master control module 1110 includes a task planning unit 1111, a task execution unit 1112, a dynamic viewpoint planning unit 1113, and a fusion map construction unit 1114. The task planning unit 1111 is used to acquire user-inputted tasks and divide the tasks into N ordered sub-tasks according to a preset task decomposition strategy. The task execution unit 1112 is used to acquire environmental perception data during the execution of the i-th sub-task and determine whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to the dynamic viewpoint planning unit 1113. The dynamic viewpoint planning unit 1113 is used to obtain a three-dimensional semantic map of the current environment from the fusion map construction unit 1114 when the collaborative operation request is received; based on the spatial position and geometric features of the operation blind spot in the three-dimensional semantic map, combined with the motion capability constraints and environmental safety conditions of the slave robot 1200, it generates a collaborative control command; the first collaborative communication module 1120 is used to send the collaborative control command to the slave robot 1200; the slave robot 1200 is used to move to the target position and adjust itself to the target posture according to the collaborative control command to obtain a regional image of the operation blind spot; and send the regional image to the fusion map construction unit 1114; the fusion map construction unit 1114 is used to perform panoramic fusion of the regional image and the three-dimensional semantic map to obtain a panoramic fusion map. This application realizes the depth perception fusion and physical collaborative operation of the master robot 1100 and the slave robot 1200, and realizes the extension of the perception range and the improvement of the operation accuracy of the heterogeneous robot collaborative operation system 1000.

[0122] Example 2

[0123] In addition, please see Figure 5 This application embodiment also provides a heterogeneous robot collaborative operation method, applied to the heterogeneous robot collaborative operation system 1000 described in the first aspect, the method including: steps S510~S560.

[0124] Step S510: The task planning unit 1111 obtains the user-inputted task and splits the task into N ordered sub-tasks according to a preset task decomposition strategy.

[0125] In step S520, the task execution unit 1112 obtains the environmental perception data during the execution of the i-th job subtask, and determines whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to the dynamic viewpoint planning unit 1113.

[0126] Step S530: When the collaborative operation request is received, the dynamic viewpoint planning unit 1113 obtains the three-dimensional semantic map of the current environment from the fusion map construction unit 1114; based on the spatial position and geometric features of the operation blind spot in the three-dimensional semantic map, combined with the motion capability constraints of the operation robot 1200 and the environmental safety conditions, a collaborative control command is generated.

[0127] In step S540, the first collaborative communication module 1120 sends the collaborative control command to the slave robot 1200;

[0128] Step S550: The work robot 1200 moves to the target position according to the cooperative control command and adjusts itself to the target posture to obtain the area image of the operation blind zone; and sends the area image to the fusion map construction unit 1114.

[0129] In step S560, the fusion map construction unit 1114 performs panoramic fusion of the regional image and the three-dimensional semantic map to obtain a panoramic fusion map.

[0130] The heterogeneous robot collaborative operation method provided in this embodiment of the invention is applied to the heterogeneous robot collaborative operation system 1000 provided in Embodiment 1 above. To avoid repetition, it will not be described again here.

[0131] This application realizes the deep perception fusion and physical collaborative operation of the master robot 1100 and the slave robot 1200, and realizes the extension of the perception range and the improvement of the operation accuracy of the heterogeneous robot collaborative operation system 1000.

[0132] Example 3

[0133] Furthermore, this embodiment of the invention provides an electronic device 600, including a memory 603 and a processor 602. The memory 603 stores a computer program, and the computer program executes the heterogeneous robot collaborative operation method provided in Embodiment 2 when it runs on the processor 602.

[0134] For details, please see Figure 6 The electronic device 600 includes: a transceiver 601, a bus interface and a processor 602. The processor 602 is used by the task planning unit 1111 to obtain the user-inputted task and split the task into N ordered sub-tasks according to a preset task decomposition strategy.

[0135] Task execution unit 1112 acquires environmental perception data during the execution of the i-th subtask and determines whether there is an operation blind spot based on the environmental perception data. If there is, it sends a collaborative operation request to dynamic viewpoint planning unit 1113. When the collaborative operation request is received, dynamic viewpoint planning unit 1113 acquires a three-dimensional semantic map of the current environment from fusion map construction unit 1114. Based on the spatial position and geometric features of the operation blind spot in the three-dimensional semantic map, combined with the motion capability constraints of the slave robot 1200 and environmental safety conditions, a collaborative control command is generated. First collaborative communication module 1120 sends the collaborative control command to slave robot 1200. Slave robot 1200 moves to the target position according to the collaborative control command and adjusts itself to the target posture to acquire the area image of the operation blind spot. It sends the area image to fusion map construction unit 1114. Fusion map construction unit 1114 performs panoramic fusion of the area image and the three-dimensional semantic map to obtain a panoramic fusion map.

[0136] In this embodiment of the invention, the electronic device 600 further includes a memory 603. Figure 6 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors 602 (represented by processor 602) and memory 603 (represented by memory 603). The bus architecture can also link various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 601 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. Processor 602 is responsible for managing the bus architecture and general processing, and memory 603 can store data used by processor 602 during operation.

[0137] The electronic device 600 provided in this embodiment of the invention can execute the heterogeneous robot collaborative operation method provided in the above-described method embodiment 2. To avoid repetition, it will not be described again here.

[0138] Example 4

[0139] Furthermore, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor 602, implements the heterogeneous robot collaborative operation method provided in Embodiment 2.

[0140] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0141] The computer-readable storage medium provided in this embodiment can implement the heterogeneous robot collaborative operation method provided in Embodiment 2. To avoid repetition, it will not be described again here.

[0142] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0143] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0144] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A heterogeneous robot collaborative work system, characterized by, The system comprises a master work robot and a slave work robot; The master work robot comprises a master control module and a first cooperative communication module, the master control module comprises a task planning unit, a task execution unit, a dynamic viewpoint planning unit and a fusion map construction unit; The task planning unit is configured to obtain a work task input by a user, and split the work task into N ordered work sub-tasks according to a preset task decomposition strategy; The task execution unit is configured to obtain environmental perception data in an execution process of an i-th work sub-task, and determine whether there is an operation blind area according to the environmental perception data, and if there is, send a cooperative work request to the dynamic viewpoint planning unit; The dynamic viewpoint planning unit is configured to obtain a three-dimensional semantic map of a current environment from the fusion map construction unit when the cooperative work request is received; generate a cooperative control instruction based on a spatial position and geometric features of the operation blind area in the three-dimensional semantic map, in combination with motion capability constraints of the slave work robot and environmental safety conditions; The first cooperative communication module is configured to send the cooperative control instruction to the slave work robot; The slave work robot is configured to move to a target position and adjust itself to a target attitude according to the cooperative control instruction, so as to obtain a regional image of the operation blind area; and send the regional image to the fusion map construction unit; The fusion map construction unit is configured to perform panoramic fusion on the regional image and the three-dimensional semantic map to obtain a panoramic fusion map; The master control module further comprises a physical cooperative management unit; The task execution unit is further configured to obtain operation precision in the execution process of the i-th work sub-task, and if the operation precision is lower than a preset precision threshold, send a support work request to the physical cooperative management unit; The physical cooperative management unit is configured to generate a support control instruction based on an operation pose and an expected force direction of the master work robot included in the i-th work sub-task when the support work request is received; The first cooperative communication module is further configured to send the support control instruction to the slave work robot; The slave work robot is configured to move to a target support position and adjust itself to a target support attitude according to the support control instruction, and then enter an attitude locking mode to provide physical support for the master work robot to execute the i-th work sub-task.

2. The heterogeneous robotic collaborative work cell of claim 1, wherein, The master work robot further comprises an interaction module, and the interaction module comprises a rendering unit; The rendering unit is configured to obtain the panoramic fusion map from the fusion map construction unit, and perform image rendering according to the panoramic fusion map to obtain a visual rendering image for representing the current environment.

3. The heterogeneous robotic collaborative work cell of claim 2, wherein, The slave work robot comprises a second cooperative communication module and a motion control module; The second cooperative communication module is configured to obtain the cooperative control instruction from the first cooperative communication module, analyze the cooperative control instruction to obtain a plurality of control parameters, and send a motion control signal to the motion control module according to each control parameter. The motion control module is configured to control the slave work robot to move to a target position and adjust to a target pose according to the control signal.

4. The heterogeneous robot collaborative job system of claim 3, wherein, The slave work robot further comprises an image acquisition module. The image acquisition module is configured to acquire the area image of the operation blind area and send the area image to the second cooperative communication module. The second cooperative communication module is further configured to send the area image to the first cooperative communication module.

5. The heterogeneous robotic collaborative work cell of claim 4, wherein, The panoramic fusion of the area image and the three-dimensional semantic map to obtain a panoramic fusion map comprises: acquiring a fixed transformation matrix of the image acquisition module and the interaction module pre-calibrated; acquiring real-time poses of the master work robot and the slave work robot; constructing a coordinate transformation chain based on the fixed transformation matrix and the real-time poses; calculating a projection relationship of the area image in a three-dimensional semantic map coordinate system according to the coordinate transformation chain; superimposition fusion of the area image and the three-dimensional semantic map according to the projection relationship to obtain the panoramic fusion map.

6. The heterogeneous robotic collaborative work cell of claim 5, wherein, Before the superimposition fusion of the area image and the three-dimensional semantic map according to the projection relationship, the method further comprises: time synchronization processing of the area image.

7. A heterogeneous robot cooperative work method characterized by, The method is applied to the heterogeneous robot cooperative work system of any one of claims 1-6, and the method comprises: a task planning unit acquires a work task input by a user and splits the work task into N ordered work sub-tasks according to a preset task decomposition strategy; a task execution unit acquires environmental perception data in an i-th work sub-task execution process and determines whether there is an operation blind area according to the environmental perception data, and if there is, sends a cooperative work request to a dynamic viewpoint planning unit; when the cooperative work request is received, the dynamic viewpoint planning unit acquires a three-dimensional semantic map of a current environment from a fusion map construction unit; generates a cooperative control instruction based on a spatial position and geometric features of the operation blind area in the three-dimensional semantic map, in combination with motion capability constraints of the slave work robot and environmental safety conditions; a first cooperative communication module sends the cooperative control instruction to the slave work robot; the slave work robot moves to a target position and adjusts to a target pose according to the cooperative control instruction to acquire an area image of the operation blind area; and sends the area image to a fusion map construction unit; the fusion map construction unit performs panoramic fusion of the area image and the three-dimensional semantic map to obtain a panoramic fusion map.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the heterogeneous robot cooperative work method of claim 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the heterogeneous robot cooperative work method of claim 7.

Images