Multi-heterogeneous robot system collaborative formation hardware test platform

Abstract

This invention provides a hardware testing platform for collaborative formation of multiple heterogeneous robot systems, relating to the field of visual robots. The testing platform includes low-cost components such as network switches, controller hosts, and RGB camera groups, enabling high-precision dynamic capture and personalized robot control of multiple heterogeneous robots. The accompanying virtual twin system can perform high-precision 3D modeling and visualization mapping of the real environment and achieve bidirectional data interaction with the physical hardware system, for efficient verification of formation planning methods and path planning algorithms.

Description

Technical Field

[0001] This invention relates to the field of robotics technology, and more specifically, to a hardware testing platform for collaborative formation of multi-heterogeneous robot systems. Background Technology

[0002] In recent years, multi-robot collaborative systems have shown broad application prospects in industries, agriculture, and rescue due to their advantages such as improved fault tolerance and environmental adaptability, becoming a research hotspot. Among the key issues are the orderly formation control, high-precision positioning, and optimized planning of multiple robots. Common formation control methods include the virtual structure method, the behavior-based method, and the leader-follower method. The leader-follower method, by planning the path of the leader robot and determining the relative positions of the follower robots based on the leader robot's motion state, is relatively simple to implement and has good scalability, making it a key research direction.

[0003] However, existing formation control, localization, and planning algorithms face numerous challenges when applied to heterogeneous multi-robot systems. The differences in the shapes of heterogeneous robots make identification difficult, potentially leading to confusion among robots in the formation; at the same time, due to the lack of fine-grained pose adjustment mechanisms, multi-robot formations often appear disordered.

[0004] In addition, validating the effectiveness of an algorithm requires building a testing platform. However, existing commercial multi-robot testing systems typically use expensive dedicated hardware, such as high-precision motion capture systems, which result in high deployment and usage costs and low testing and evaluation efficiency. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a hardware testing platform for collaborative formation of multiple heterogeneous robot systems, comprising a network switch, a controller host, and an RGB camera group. Multiple heterogeneous robots, the RGB camera group, and the controller host are connected via the network switch. The RGB camera group is used to acquire image data of the multiple heterogeneous robots in a scene in real time. The controller host includes a multi-robot collaborative formation control system, which comprises a visual positioning subsystem, a robot path planning subsystem, and a multi-robot formation subsystem. The visual positioning subsystem performs multi-target tracking and pose analysis based on the image data to determine the target category, target category ID, and target pose of the heterogeneous robots. The multi-robot formation subsystem determines the formation mode, formation role, and motion constraint information of the multiple heterogeneous robots, and receives the desired robot motion path output by the robot path planning subsystem and outputs control commands to control the multiple heterogeneous robots based on the desired robot motion path. The robot path planning subsystem receives the motion constraint information and the target pose, and generates the desired robot motion path according to a preset path planning algorithm.

[0006] The low-cost multi-heterogeneous robot system collaborative formation hardware test platform provided in this invention integrates low-cost hardware components such as RGB camera groups, network switches, and controller hosts to construct a test environment that supports multiple heterogeneous robots. It effectively solves the problem of high cost of traditional high-precision robot dynamic capture systems, and provides economical and practical hardware and software support for the development of multi-robot collaborative technologies such as formation, localization, and planning. It provides a low-cost and efficient solution for the development of heterogeneous multi-robot collaborative technology, which has important theoretical significance and application value, and promotes the progress and widespread application of technology in this field.

[0007] Optionally, the controller host further includes a twin system; the twin system is connected to the visual positioning subsystem, the robot path planning subsystem, and the multi-robot formation subsystem respectively; the twin system is used to receive the target category, target category ID, target pose, and desired motion path of the heterogeneous robot, and to perform real-time three-dimensional visualization mapping based on the target category, target category ID, target pose, and desired motion path of the robot.

[0008] This invention innovatively integrates a twin system, providing a high-performance verification platform for algorithm development. It enables rapid evaluation of the effectiveness of new algorithms, significantly shortening the development cycle. Seamless integration between real and virtual environments is achieved, allowing the verified optimal algorithm model to be directly applied to the physical system, avoiding repeated debugging and improving system reliability. The aforementioned testing platform is applicable to diverse testing environments, supporting simulation testing of complex heterogeneous robot scenarios, thus improving the robustness and applicability of the algorithm.

[0009] Optionally, the twin system includes an interaction system and a twin communication module. The interaction system includes a digital modeling library and a visualization interaction module. The twin communication module is used to connect the interaction system and each of the subsystems. The digital modeling library stores three-dimensional model data of the heterogeneous robot and its motion trajectory in a real scene. The visualization interaction module is used to provide a control interface for receiving user control commands and to perform real-time three-dimensional visualization mapping of the real scene and the heterogeneous robot in it, the robot's expected motion path, and the actual motion path.

[0010] In this embodiment of the invention, the digital modeling library can pre-store 3D model data of robots and their motion trajectories in real scenes, which can be easily accessed at any time; the visualization module is a human-computer interaction visualization interface, which allows users to view the 3D visualization mapping of the real scene environment and the robot's motion state in the virtual space in real time.

[0011] Optionally, the twin system is used to output twin motion data of each of the heterogeneous robots for error analysis with the real-time motion data of each of the heterogeneous robots.

[0012] In this embodiment of the invention, the twin system outputs the aforementioned twin motion data. By analyzing the error between the twin motion data and the real-time motion data, the error of the aforementioned test platform can be analyzed, and its effectiveness and feasibility can be verified.

[0013] Optionally, the robot path planning subsystem is used to receive virtual environment information and path planning algorithm feedback from the interactive system, and to update the robot's desired motion path based on the feedback virtual environment information and path planning algorithm.

[0014] In this embodiment of the invention, users can input virtual environment information and path planning algorithms based on the interactive system to flexibly adjust the testing process.

[0015] Optionally, the twin system includes a formation planning algorithm model library and a path planning algorithm model library.

[0016] In this embodiment of the invention, the twin system has a rich built-in array planning algorithm model library and path planning algorithm model library, providing a high-performance verification platform for new algorithms.

[0017] Optionally, the visual positioning subsystem includes a data initialization module, a neural network module, a multi-target tracking module, and a pose analysis module. The data initialization module preprocesses the image data acquired by the RGB camera group and sends the processed data to the neural network module. The neural network module performs target detection and keypoint detection on the preprocessed data using a weight file, and outputs the category and keypoint information to the multi-target tracking module and the pose analysis module. The multi-target tracking module combines a prior motion model and a data association algorithm to identify the category and keypoint information to obtain the target category and target category ID of each target object. The pose analysis module calculates the 6-DOF pose of each target in three-dimensional space based on the keypoint information and the intrinsic and extrinsic parameters of the RGB camera.

[0018] The visual positioning subsystem in this embodiment of the invention can achieve accurate identification and tracking of the target as well as pose recognition.

[0019] Optionally, the multi-robot formation subsystem includes a formation planning module, a navigator-follower formation module, and a robot control module. The formation planning module assigns navigator or follower roles to the multiple heterogeneous robots based on a pre-set formation strategy or a formation strategy developed in real-time from the twin system, combined with target information output by the visual positioning subsystem. It also provides motion constraint information for the navigator to the navigator-follower formation module. The navigator-follower formation module determines the desired position and formation of each heterogeneous robot and outputs it to the robot path planning subsystem. The robot control module receives the desired path output by the robot path planning subsystem, decomposes it into motion control commands for each heterogeneous robot, and sends them to each heterogeneous robot for execution.

[0020] The multi-robot formation subsystem in this embodiment of the invention can realize path planning for multiple heterogeneous robots.

[0021] Optionally, the robot control module is further configured to share the status information and control commands of each heterogeneous robot with the twin system.

[0022] In this embodiment of the invention, the hardware testing platform integrates a twin system, which can share the status information and control commands of heterogeneous robots and perform visual mapping.

[0023] Optionally, the preset path planning algorithm is a fast expanding random tree algorithm.

[0024] The present invention provides a feasible preset path planning algorithm, which has been verified to be effective by a hardware testing platform. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 A detailed structural block diagram of a low-cost hardware testing platform for verifying formation, localization, and planning methods for heterogeneous multi-robot systems, provided in an embodiment of the present invention;

[0027] Figure 2 This is a three-dimensional simulation model of a robot based on a twin system in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the experimental environment and equipment for Embodiment 1 of the present invention;

[0029] Figure 4 This is a schematic diagram of a fixed path for multiple robots in a real environment, as given in Embodiment 1 of the present invention;

[0030] Figure 5 This is a schematic diagram of the multi-robot motion process observed in a real-world twin system according to Embodiment 1 of the present invention.

[0031] Figure 6 This is a schematic diagram illustrating the error between real data and twin data in Embodiment 1 of the present invention;

[0032] Figure 7 This is a schematic diagram of the RRT algorithm.

[0033] Figure 8 This is a schematic diagram of the actual environment and the initial state of the robot in Embodiment 2 of the present invention;

[0034] Figure 9 This is a schematic diagram of path planning generated based on a twin system in an embodiment of the present invention;

[0035] Figure 10 This is a schematic diagram illustrating the process of a robot moving along a path generated by a twin platform in a real environment, as described in Embodiment 2 of the present invention.

[0036] Figure 11 This is a schematic diagram of the initial state of the virtual environment in the twin system in Embodiment 3 of the present invention;

[0037] Figure 12 A schematic diagram of the leader-follower movement model;

[0038] Figure 13 This is a schematic diagram of the complete process of multi-heterogeneous robot formation collaboration based on this platform in Embodiment 3 of the present invention, which includes multi-heterogeneous robot formation path planning, multi-heterogeneous robot 6D pose analysis, multi-robot path planning, and changing the orientation of multi-heterogeneous robots. Detailed Implementation

[0039] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0040] In view of the above-mentioned problems existing in the prior art, the present invention provides a low-cost hardware testing platform for collaborative formation of multi-heterogeneous robot systems.

[0041] On the one hand, the aforementioned hardware platform includes low-cost components such as network switches, controller hosts, and RGB (Red, Green, Blue) camera arrays, enabling high-precision dynamic capture and personalized robot control for multiple heterogeneous robots. On the other hand, the accompanying virtual twin system can perform high-precision 3D modeling and visualization mapping of the real environment, and achieve bidirectional data interaction with the physical hardware system for efficient verification of formation planning methods and path planning algorithms.

[0042] The hardware testing platform for collaborative formation of multiple heterogeneous robot systems provided in this embodiment of the invention includes a network switch, a controller host, and an RGB camera group. Multiple heterogeneous robots, the RGB camera group, and the controller host exchange data through the aforementioned network switch.

[0043] Specifically, the aforementioned RGB camera group is used to acquire image data from multiple heterogeneous robots within the scene in real time. This RGB camera group can include multiple RGB cameras positioned at different locations, capable of acquiring image data from the scene in real time for analysis by the controller host.

[0044] The controller host serves as the processing center of the test platform, receiving real-time data streams from the network switch and integrating a multi-robot collaborative formation control system and a communication module system. The multi-robot collaborative formation control system comprises a vision positioning subsystem, a robot path planning subsystem, and a multi-robot formation subsystem. The three subsystems are detailed below:

[0045] The visual positioning subsystem is used to perform multi-target tracking and pose analysis based on the above image data, and to determine the target category, target category ID (Identity Document), and target pose of the heterogeneous robot.

[0046] The main function of this visual positioning subsystem is to process the scene RGB real-time information acquired from the RGB camera group through the aforementioned communication module system, including multi-target tracking and pose analysis, and output the target category, target ID, and target 6D pose information in the scene. Simultaneously, the communication module system can share the target category, target ID, and target 6D pose information with the twin system.

[0047] For example, the visual positioning subsystem described above may include a data initialization module, a neural network module, a multi-target tracking module, and a pose analysis module.

[0048] The data initialization module preprocesses the image data acquired by the RGB camera group and sends the processed data to the neural network module. Specifically, it preprocesses the raw RGB camera data from the communication module system and sends the processed data to the neural network module to meet the data format requirements of subsequent neural network processing.

[0049] The neural network module performs object detection and keypoint detection on preprocessed data using a weight file, and outputs the category and keypoint information to the multi-object tracking module and pose analysis module.

[0050] The multi-object tracking module combines prior motion models and data association algorithms to identify the category and key point information of each target object, thus obtaining the target category and target category ID. By combining prior motion models and data association algorithms, the multi-object tracking module further analyzes the category and key point information output by the neural network to accurately identify the ID of each target object.

[0051] The pose analysis module is used to calculate the 6-DOF pose of each target in 3D space based on keypoint information and the intrinsic and extrinsic parameters of the RGB camera. Specifically, based on the aforementioned keypoint information and the camera's intrinsic and extrinsic parameters, the 6-DOF pose of each target in 3D space is calculated.

[0052] The multi-robot formation subsystem is used to determine the formation mode, formation role, and motion constraint information of multiple heterogeneous robots, as well as to receive the desired robot motion path output by the robot path planning subsystem and output control commands to control multiple heterogeneous robots based on the desired robot motion path.

[0053] The main function of this multi-robot formation subsystem is to process information from the aforementioned visual positioning subsystem, including target category, target category ID, and target 6D pose, to determine the cooperative formation mode, formation roles, and motion constraints of the multiple robots. These motion constraints include initial positions and formation boundary conditions. The system provides the path planning subsystem with the coordinates of the lead robot's position, receives the desired path from the path planning system, and further outputs control commands to the communication module to control the movement of each heterogeneous robot. The heterogeneous robots move based on these control commands, efficiently testing formation, localization, and planning technologies. Simultaneously, the communication module system can share the robots' real-time status data with the twin system.

[0054] Specifically, the multi-robot formation subsystem may include a formation planning module, a navigation-following formation module, and a robot control module.

[0055] The formation planning module assigns navigator or follower roles to multiple heterogeneous robots based on pre-set formation strategies or real-time formation strategies from the twin system, combined with target information output by the visual positioning subsystem, and provides motion constraint information of the navigator to the navigator-follower formation module.

[0056] The navigator-follower formation module is used to determine the desired positions and formation patterns of each heterogeneous robot, and outputs this information to the robot path planning subsystem. Based on the output of the formation planning module, and combined with the robot's kinematic and dynamic models, the relative motion states and formation geometry of the navigator and followers are planned, outputting the desired position and formation pattern of the navigator robot. This information serves as the starting point and boundary conditions for the robot path planning subsystem, which is then output to the robot path planning module.

[0057] The robot control module receives the desired path output by the robot path planning subsystem, decomposes it into motion control commands for each heterogeneous robot, and sends them to each robot for execution. The robot control module receives the desired path output by the robot path planning subsystem, combines the robot's inverse kinematics mechanism and controller, decomposes it into corresponding motion control commands for each robot, and sends them to each robot for execution via the communication module system. Simultaneously, the robot's real-time status information and control commands can be shared with the twin system through the communication module system.

[0058] The robot path planning subsystem receives the aforementioned motion constraint information and target pose, and generates the robot's desired motion path according to a preset path planning algorithm. This subsystem processes the starting point and boundary condition information from the multi-robot formation subsystem, as well as the target 6D pose information and environmental information provided by the visual positioning subsystem. Through a pre-set or real-time planned path planning algorithm, it generates the robot's desired motion path for the robot control module.

[0059] For example, the preset path planning algorithm is the Rapidly Exploring Random Tree (RRT) algorithm.

[0060] The aforementioned network switch serves as a data exchange hub, providing communication support and data transmission between multiple heterogeneous robots, cameras, and controller hosts via wired or wireless means.

[0061] A hardware testing platform for collaborative formation of heterogeneous robot systems, built upon low-cost components such as network switches, controller hosts, and RGB camera groups, can acquire real-time image data of the scene via the RGB camera group. The visual positioning subsystem identifies the category and pose of the heterogeneous robots, while the multi-robot formation subsystem determines the formation mode, roles, and motion constraints of the robots for testing. The robot path planning subsystem performs path planning based on the algorithm under test. Furthermore, the multi-robot formation subsystem decomposes the desired motion path into motion control commands for each robot, issuing them separately for execution. This allows observation of the actual movement of each robot under the aforementioned formation and path planning conditions, efficiently verifying key technologies such as heterogeneous multi-robot formation, localization, and planning. During multi-robot movement, the visual positioning subsystem continuously tracks targets and analyzes poses, adjusting them promptly to ensure orderly formation.

[0062] The embodiments of this invention employ inexpensive, general-purpose hardware components such as controller hosts, network switches, and RGB cameras, resulting in low system deployment costs and lowering the barrier to commercial use. Through a visual positioning subsystem, a robot path planning subsystem, a multi-robot formation subsystem, and an RGB camera group, accurate identification and pose adjustment are achieved during the formation of heterogeneous robots, ensuring orderly formation.

[0063] The low-cost multi-heterogeneous robot system collaborative formation hardware test platform provided in this invention integrates low-cost hardware components such as RGB camera groups, network switches, and controller hosts to construct a test environment that supports multiple heterogeneous robots. It effectively solves the problem of high cost of traditional high-precision robot dynamic capture systems, and provides economical and practical hardware and software support for the development of multi-robot collaborative technologies such as formation, localization, and planning. It provides a low-cost and efficient solution for the development of heterogeneous multi-robot collaborative technology, which has important theoretical significance and application value, and promotes the progress and widespread application of technology in this field.

[0064] Furthermore, the aforementioned testing platform is also equipped with a virtual twin system, which not only enables digital modeling and visual mapping of the real environment, but also provides a high-performance verification platform for formation planning algorithms, localization algorithms, and path planning algorithms. Based on this, the aforementioned controller host also includes a twin system; this twin system is connected to the visual localization subsystem, the robot path planning subsystem, and the multi-robot formation subsystem, respectively.

[0065] The twin system can receive the target category, target category ID, target pose, and expected motion path of the heterogeneous robot, and perform real-time 3D visualization mapping based on the target category, target category ID, target pose, and expected motion path of the robot.

[0066] Specifically, this embodiment uses a twin system to map the hardware platform in real time. The visual positioning subsystem can share target category, target ID, and target 6D pose information with the twin system through the communication module system. The multi-robot formation subsystem can share real-time robot status data and control commands with the twin system through the communication module system. The robot path planning subsystem interacts with the twin system, updating the path planning strategy in a timely manner based on the virtual environment information fed back by the twin system and the human-computer interaction system. It also visualizes the path planning results and maps them to the virtual environment of the twin system, allowing users to intuitively and clearly view the desired path.

[0067] This invention innovatively integrates a twin system, providing a high-performance verification platform for algorithm development. It enables rapid evaluation of the effectiveness of new algorithms, significantly shortening the development cycle. Seamless integration between real and virtual environments is achieved, allowing the verified optimal algorithm model to be directly applied to the physical system, avoiding repeated debugging and improving system reliability. The aforementioned testing platform is applicable to diverse testing environments, supporting simulation testing of complex heterogeneous robot scenarios, thus improving the robustness and applicability of the algorithm.

[0068] For example, the above-mentioned twin system includes an interactive system and a twin communication module. The interactive system includes a digital modeling library and a visualization interactive module. The twin communication module is used to connect the interactive system and each subsystem.

[0069] The digital modeling library stores 3D model data of heterogeneous robots and their motion trajectories in real-world scenarios. This library can store 1:1 3D model data of robots and their motion trajectories in real-world scenarios, including robot motion parameters, sensor parameters, and appearance parameters, which can be accessed by the interactive system at any time.

[0070] The visualization and interaction module provides a control interface for receiving user commands and performs real-time 3D visualization mapping of the real scene, its heterogeneous robots, the robots' expected motion paths, and their actual motion paths. This visualization module serves as a human-computer interaction interface, enabling users to view a real-time 3D visualization mapping of the real-world environment and the robots' motion states within it in a virtual space.

[0071] The twin communication module provides a data transmission bus for the twin simulation platform to achieve bidirectional data exchange with the communication module in the controller host, thereby mapping the state of the real environment to the virtual space in real time, and reflecting the planning instructions of the virtual space to the real environment.

[0072] Optionally, the twin system can also output twin motion data for each heterogeneous robot to perform error analysis with the real-time motion data of each heterogeneous robot. On one hand, the twin system can perform real-time twinning based on the analysis results obtained from image data acquired by the RGB camera group; on the other hand, the controller host can obtain the robot's real-time motion data (including robot sensor feedback data) through the robot operating system. The twin system outputs the aforementioned twin motion data, and by analyzing the error between the twin motion data and the real-time motion data, the error of the aforementioned test platform can be analyzed, verifying its effectiveness and feasibility. Verification shows that the aforementioned twin system can effectively capture the motion of robots in real-world scenes using ordinary RGB cameras and perform virtual simulation within the twin system, achieving low error even with low-cost RGB cameras.

[0073] Users can input virtual environment information and path planning algorithms (including adjusting algorithm parameters) through the interactive system. The robot path planning subsystem can receive the information fed back by the interactive system and update the robot's desired motion path based on the feedback virtual environment information and path planning algorithm.

[0074] Optionally, the aforementioned twin system includes a rich library of formation planning and path planning algorithm models, providing a high-performance verification platform for new algorithms. The formation planning algorithm model library can include various multi-robot formation control algorithm models, such as navigation-following methods, behavior-based methods, virtual structure methods, graph theory-based methods, artificial potential field methods, and proportional-integral-derivative control; the path planning algorithm model library can include various path planning algorithms, such as the fast expanding random tree algorithm, ant colony algorithm, etc.

[0075] Genetic algorithms, etc.

[0076] The visualization and interaction module provides users with an intuitive human-computer interaction interface, enabling them to view simulation effects, adjust algorithm parameters, and compare different solutions.

[0077] Figure 1 This is a detailed structural block diagram of a low-cost hardware test platform for verifying formation, localization, and planning methods for heterogeneous multi-robot systems, provided as an embodiment of the present invention. Figure 1 As shown, a low-cost test hardware platform for verifying the formation, localization and planning methods of heterogeneous multi-robot systems consists of an RGB camera group (RGB camera 1 to RGB camera n), a network switch and a controller host. It constructs a hardware test platform that supports multiple heterogeneous robots (robot 1 to robot n), a twin system integrated by the controller host, a multi-robot cooperative formation control system and a communication module system, which constitute a virtual twin simulation platform.

[0078] The hardware test platform includes a controller host, network switches, and an RGB camera array. The controller host serves as the processing center of the test platform, integrating a twin system, a multi-robot collaborative formation control system, and a communication module system.

[0079] Network switches serve as data exchange hubs, providing communication support and data transmission for multiple robots, cameras, and control hosts via wired or wireless means.

[0080] An RGB camera group consists of one or more RGB cameras, used to acquire RGB image data in the scene in real time, and transmit the acquired data to the controller host for analysis and processing via a network switch.

[0081] The twin system includes a digital modeling library and a visualization and interaction module. The digital modeling library constructs high-precision 3D digital models of multiple robots and their environments in real-world scenarios, providing data support for the virtual simulation environment. Figure 2 This is a three-dimensional simulation model of a robot based on a twin system in an embodiment of the present invention. Figure 2 'a' represents a 3D simulation model of a typical Mecanum wheel robot. Figure 2 b is a 3D simulation model of the four-axis robotic arm Mecanum wheel robot. Figure 2 c represents the 3D simulation model of the two-wheel differential robot. Figure 2 d represents the three-dimensional simulation model of the three-wheeled transport robot.

[0082] The twin system interacts bidirectionally with the communication module system through a twin communication module, achieving real-time data mapping between the virtual and real environments and ensuring a high degree of consistency between them. The twin system incorporates a rich library of formation planning and path planning algorithm models, providing a high-performance verification platform for new algorithms. The visualization and interaction module offers users an intuitive human-computer interface, enabling them to view simulation results, adjust algorithm parameters, and compare different solutions.

[0083] The multi-robot cooperative formation control system includes a vision positioning subsystem, which processes real-time RGB image data of the scene acquired by the RGB camera group through the communication module system, outputs the category, ID, and six-dimensional pose information of each target object in the scene, and shares this information with the twin system in real time through the communication module system. The multi-robot cooperative formation control system also includes a multi-robot formation subsystem and a robot path planning subsystem.

[0084] The communication module system provides a data exchange path for the twin system, the multi-robot collaborative formation control system, and external hardware (including network switches, multiple robots, and RGB camera groups), enabling data interaction between the above modules.

[0085] based on Figure 1The hardware testing platform shown is described in detail in the following embodiments.

[0086] Example 1: Verifying the error between the vision-based real-time twin data and the real environment data in this invention.

[0087] This embodiment uses a four-axis robotic arm Mecanum wheel robot as the navigator and two isomorphic ordinary Mecanum wheel robots as followers for the experiment. Figure 3 This is a schematic diagram of the experimental environment and equipment for Embodiment 1 of the present invention. Figure 3 The image shows the navigator in the middle, the followers 1 and followers on the top and bottom sides, a 1.5m height camera, and the ROS (Robot Operating System) installed on the laptop.

[0088] By setting up robot formation models and motion models, establishing fixed formation paths, and collecting real-time robot motion data, the process is simulated using a twin system. Data collected in real-time by RGB cameras is analyzed. The twin system records the motion trajectory parameters of the vision-based positioning subsystem. Error analysis is performed between the real-time data and the real-time twin analysis data of this platform's twin system to verify the effectiveness and feasibility of the visual positioning subsystem of this test platform. This verification process includes the following steps:

[0089] Step 1: Camera Calibration - Obtain Camera Parameters

[0090] For example, the camera is 1.5m high and has a field of view of 2.4m x 2.4m.

[0091] Step 1-1, camera calibration refers to the calibration of the camera's lens parameters and the calibration of the camera relative to various positions in the scene;

[0092] Steps 1-2: The camera calibration method uses the Zhang Zhengyou calibration method with a checkerboard. The obtained camera parameters are used to transform the camera coordinate system to the world coordinate system.

[0093] Step 2: Configure the environment and simulation parameters in the twin system

[0094] The camera parameters were configured using the twin system, including setting the camera height in the simulation environment to 1.5m and the camera's field of view to 2.4m x 2.4m; high-precision 3D models of the aforementioned four-axis robotic arm Mecanum wheel robot and a regular Mecanum wheel robot were constructed using the twin system's digital modeling library and imported into the virtual simulation environment, such as... Figure 2 a, Figure 2 As shown in b; in the twin system of this platform, key points are labeled on the robot, and the neural network is trained to obtain a weight file for real-time analysis by the vision system.

[0095] Step 3: Configure the hardware platform

[0096] The network switch is wirelessly connected to a four-axis Mecanum wheel robot and a regular Mecanum wheel robot, and communicates with the controller host. The camera is also connected to the network switch to transmit real-time RGB image data to the controller host. Figure 3 As shown; independent of this test platform, the robots are controlled wirelessly through the underlying ROS system to form a formation, and the formation robots are controlled to move along a fixed path, as shown. Figure 4 As shown. Figure 4 This is a schematic diagram of a fixed path for multiple robots in a real environment, as shown in Example 1.

[0097] Step 4: Begin the hardware experiment

[0098] In real-world scenarios, the twin system based on this platform allows users to view real-time camera recordings of the motion of multiple heterogeneous robot formations following a given path. The ROS system is then used to acquire and export the real-time motion data of the multi-robot formation. Figure 5 This is a schematic diagram of the multi-robot motion process observed in a real-world twin system, as shown in Example 1. Figure 5 The process of multiple robots moving along a given path is illustrated. The twin motion data of multiple heterogeneous robot formations based on the visual positioning subsystem is exported from the twin system of this platform, and the real-time motion data of the above multi-robot formations and the twin motion data of the multi-heterogeneous robot formations are analyzed.

[0099] Step 5: Error Analysis

[0100] During the formation process, there is a certain error between the multi-robot formation twin motion data exported by the twin system of this platform and the real-time multi-robot motion data in the actual ROS system. Figure 6 This diagram illustrates the error between real data and twin data in Example 1. Figure 6 a is the X-axis error graph. Figure 6 b is the Y-axis error graph.

[0101] like Figure 6 As shown, at 10s, 27s, and 32s, the navigator robot undergoes significant changes in motion, and the followers experience certain positional errors due to communication delays and their own inertia. However, the system stabilizes within a short period of time, and the twin system restores the real-time motion data of the multi-robot formation to accurate dynamic capture.

[0102] The twin system of this platform can effectively capture the motion of robots in real-world scenes using ordinary RGB cameras and perform virtual simulations within the twin system, achieving low error even with low-cost RGB cameras.

[0103] Example 2: Verifying the feasibility of a multi-robot cooperative formation path planning method based on the visual localization RRT algorithm according to the present invention.

[0104] This embodiment uses a four-axis robotic arm Mecanum wheel robot and a regular Mecanum wheel robot as examples to verify the vision-based RRT algorithm multi-robot cooperative formation path planning method on the platform of this invention.

[0105] The objective of this embodiment is to develop a four-axis Mecanum wheel robot, specifically a standard Mecanum wheel robot, that performs path planning for a PlayStation 2 controller. The robot navigates obstacles to reach both the front and rear of the controller, with each robot facing the controller, thus completing a collaborative task. First, a twin system based on this platform is used to simulate this task objective, generating the expected trajectory for the path planning process. Then, this multi-robot collaborative formation path planning method is tested and verified in a real-world environment.

[0106] Step 1: Obtain camera extrinsic parameters through camera calibration

[0107] Step 1-1, camera calibration refers to the calibration of the camera's lens parameters and the calibration of the camera relative to various positions in the scene;

[0108] Steps 1-2: The camera calibration method uses the Zhang Zhengyou calibration method with a checkerboard. The obtained camera parameters are used to transform the camera coordinate system to the world coordinate system.

[0109] Step 2: Configure the environment and simulation parameters in the twin system

[0110] Configure camera parameters through a twin system, including setting the camera height in the simulation environment to 1.5m and the camera's field of view to 2.4m*2.4m;

[0111] High-precision 3D models of the aforementioned four-axis robotic arm Mecanum wheel robot, ordinary Mecanum wheel robot, and PS2 controller were constructed using the digital modeling library of the twin system and imported into the virtual simulation environment. The 3D models are as follows: Figure 2 a, Figure 2 As shown in b.

[0112] In the twin system of this platform, key points are labeled on the robot and the neural network is trained to obtain a weight file for real-time analysis by the vision system.

[0113] The multi-robot path planning subsystem was edited using a twin system to verify the vision-based RRT algorithm for multi-robot cooperative formation path planning.

[0114] Specifically, the schematic diagram of the RRT algorithm in step 2 is as follows: Figure 7 As shown, the specific steps are as follows:

[0115] Step 2-1, in space V, with X init The (x, y, θ) pose point is used as the root node of the RRT tree;

[0116] Step 2-2: Randomly generate a pose point X. rand ;

[0117] Step 2-3: Traverse the leaf nodes of the RRT tree to find the distance X from step 2-2. rand The nearest node X near Based on the robot's motion model, select input u∈U (such as steering angle, velocity, etc.) so that the robot can move along X under the action of this input. nar Arrival at X rand ;

[0118] Steps 2-4, from X in step 2-3 near To X in step 2-2 rand Extend a distance in the direction to X new And determine X new Check if the point satisfies the no-collision constraint. If not, return to step 2-3.

[0119] Steps 2-5, X new Add to the RRT tree and set X near As its parent node, repeat this process until a new X is generated. new If the distance between a point and the target point is less than a threshold (constant), the random tree construction ends. Parent nodes are then taken sequentially from that point until X. init .

[0120] Step 3: Configure the hardware platform

[0121] The network switch is wirelessly connected to the four-axis robotic arm Mecanum wheel robot and the ordinary Mecanum wheel robot, and communicates with the controller host. The camera is connected to the network switch to transmit real-time RGB image data to the controller host, and the initial positions of multiple robots, obstacles, and PS2 controllers are set. Figure 8 This is a schematic diagram of the real environment and the initial state of the robot in Example 2. Figure 8 The left side shows Robot 1, the right side shows Robot 2, the middle shows an obstacle, and the top shows the target object (PS2 controller).

[0122] Step 4: Begin the experiment

[0123] First, based on the twin system of this platform, the multi-robot formation cooperative path planning method based on vision localization RRT algorithm is simulated and verified to generate the desired trajectory map, such as... Figure 9 The diagram shows a path planning scheme generated based on a twin system, where the trajectory bypasses obstacles and reaches both sides of the target object. Figure 9 In the diagram, robots 1 and 2 start from the lower starting point and move to the upper cooperation point. The square represents the obstacle and the ellipse represents the target object.

[0124] Secondly, the multi-robot cooperative formation path planning method based on the vision system RRT algorithm is experimentally verified in a real environment. Figure 10 This is a schematic diagram illustrating the process of the robot moving along the path generated by the twin platform in a real environment, as shown in Example 2. In a real environment, the robot can bypass obstacles to reach the vicinity of the target object and move towards the PS2 controller of the target object to achieve the predetermined target effect.

[0125] Through the above examples, it can be concluded that this platform, by constructing a virtual simulation environment, provides the algorithm with a verification scenario that closely matches the real environment, overcoming the shortcomings of relying solely on the experimental environment, such as the single scenario conditions and high data acquisition costs; it can map the dynamic changes of the experimental scenario in real time, record and visualize the motion state of multiple robots, and provide an intuitive basis for comparing and analyzing the actual performance and theoretical expectations of the algorithm, greatly simplifying the process of algorithm performance evaluation.

[0126] Example 3: Perform a complete algorithm simulation verification in a simulation environment, and adjust the algorithm based on the simulation results.

[0127] The objective of this embodiment is to simulate the complete heterogeneous robot formation and collaboration process based on the low-cost test hardware and twin platform of the heterogeneous multi-robot system formation, localization and planning method of the present invention, which includes multi-heterogeneous robot formation, multi-heterogeneous robot formation path planning, multi-heterogeneous robot 6D pose analysis, multi-robot path planning and changing the orientation of multi-heterogeneous robots.

[0128] First, configure the simulation environment: This simulation environment includes a three-wheeled transport robot, two isomorphic ordinary Mecanum wheel robots, and two isomorphic two-wheeled differential speed robots. The three-wheeled transport robot is set as the navigator, and the other robots as followers. High-precision 3D models of the above five heterogeneous robots are constructed using the digital modeling library in the platform's interactive system and imported into the virtual simulation environment. The 3D models are as follows: Figure 2 As shown.

[0129] Set up a room in the simulation environment, place an RGB camera at the top of the room, the position of the pentagram is the endpoint target of the collaborative formation path planning of multiple heterogeneous robots, and the red cube is the orientation reference target of the multiple heterogeneous robots. Figure 11 This is a schematic diagram of the initial state of the virtual environment in the twin system in Example 3.

[0130] Secondly, configure the simulation parameters: In this simulation verification, the camera is an ideal camera; set the robot circular formation: the three-wheeled transport robot is the leader, and the other heterogeneous robots are followers surrounding the leader, and the formation radius is the radius of the outer circle of the formation.

[0131] Set up the navigator movement model Setting up a follower motion model: A schematic diagram of the navigator-follower motion model is shown below. Figure 12 As shown; the path planning algorithm is set to the RRT algorithm path planning method, and the starting point is the center of the minimum circumcircle of the multi-heterogeneous robot formation;

[0132] Finally, the simulation begins: the twin system based on this platform can see the formation of multiple heterogeneous robots. Figure 13 This is a schematic diagram of the complete process of multi-heterogeneous robot formation collaboration based on this platform in Example 3, which includes multi-heterogeneous robot formation path planning, multi-heterogeneous robot 6D pose analysis, multi-robot path planning, and changing the orientation of multi-heterogeneous robots.

[0133] Specifically, multiple heterogeneous robots are arrayed in formation, such as Figure 13 a, Figure 13 As shown in b, the path planning for multi-heterogeneous robot formation is as follows: Figure 13 c. Figure 13 d、 Figure 13 As shown in e, the pose adjustment of multi-heterogeneous robots is as follows: Figure 13 f、 Figure 13 As shown in g, multi-robot path planning is as follows: Figure 13 As shown in h, the orientation of the multi-heterogeneous robot is changed as follows: Figure 13 As shown in i, this is the complete collaborative process of multi-heterogeneous robot formation.

[0134] As can be seen from the above embodiments, the low-cost testing hardware platform and twin system of the present invention can efficiently verify key technologies such as heterogeneous multi-robot formation, localization, and planning, and has strong applicability and scalability. It not only reduces the cycle and cost of algorithm development, but also ensures the reliability and stability of the final system, providing strong support for promoting technological progress in this field, and has significant theoretical and practical value.

[0135] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

[0136] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the above embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0138] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hardware testing platform for collaborative formation of multi-heterogeneous robot systems, characterized in that, It includes a network switch, a controller host, and an RGB camera group; multiple heterogeneous robots, the RGB camera group, and the controller host are connected through the network switch; The RGB camera group is used to acquire image data of the multiple heterogeneous robots in the scene in real time. The controller host includes a multi-robot collaborative formation control system, which includes a visual positioning subsystem, a robot path planning subsystem, and a multi-robot formation subsystem. The visual positioning subsystem is used to perform multi-target tracking and pose analysis based on the image data, and to determine the target category, target category ID, and target pose of the heterogeneous robot. The multi-robot formation subsystem is used to determine the formation mode, formation role, and motion constraint information of the multiple heterogeneous robots, and to receive the desired robot motion path output by the robot path planning subsystem and output control commands to control the multiple heterogeneous robots according to the desired robot motion path. The robot path planning subsystem is used to receive the motion constraint information and the target pose, and generate the robot's desired motion path according to a preset path planning algorithm. The controller host also includes a twin system; the twin system is connected to the visual positioning subsystem, the robot path planning subsystem, and the multi-robot formation subsystem, respectively; The twin system is used to receive the target category, target category ID, target pose, and expected motion path of the heterogeneous robot, and to perform real-time three-dimensional visualization mapping based on the target category, target category ID, target pose, and expected motion path of the robot. The visual positioning subsystem includes a data initialization module, a neural network module, a multi-target tracking module, and a pose analysis module. The data initialization module is used to preprocess the image data acquired by the RGB camera group and send the processed data to the neural network module; The neural network module is used to perform target detection and key point detection on the preprocessed data through the weight file, and outputs the category and key point information to the multi-target tracking module and the pose analysis module. The multi-target tracking module is used to combine a priori motion model and data association algorithm to identify the category and key point information of each target object to obtain the target category and target category ID of each target object; The pose analysis module is used to calculate the 6-DOF pose of each target in three-dimensional space based on the key point information and the intrinsic and extrinsic parameters of the RGB camera.

2. The hardware testing platform for collaborative formation of multi-heterogeneous robot systems according to claim 1, characterized in that, The twin system includes an interactive system and a twin communication module. The interactive system includes a digital modeling library and a visualization interactive module. The twin communication module is used to connect the interactive system and each of the subsystems. The digital modeling library stores three-dimensional model data of the heterogeneous robots and their motion trajectories in real-world scenarios; The visualization interaction module is used to provide a control interface for receiving user control commands, and to perform real-time three-dimensional visualization mapping of the real scene and the heterogeneous robot in it, the robot's expected motion path, and the actual motion path.

3. The hardware testing platform for collaborative formation of multi-heterogeneous robot systems according to claim 2, characterized in that, The twin system is used to output twin motion data of each of the heterogeneous robots for error analysis with the real-time motion data of each of the heterogeneous robots.

4. The hardware testing platform for collaborative formation of multi-heterogeneous robot systems according to claim 2, characterized in that, The robot path planning subsystem is used to receive virtual environment information and path planning algorithm feedback from the interactive system, and to update the robot's desired motion path based on the feedback virtual environment information and path planning algorithm.

5. The hardware testing platform for collaborative formation of multi-heterogeneous robot systems according to claim 4, characterized in that, The twin system includes a formation planning algorithm model library and a path planning algorithm model library.

6. The hardware testing platform for collaborative formation of multi-heterogeneous robot systems according to claim 1, characterized in that, The multi-robot formation subsystem includes a formation planning module, a navigation-following formation module, and a robot control module; The formation planning module assigns navigator or follower roles to the multiple heterogeneous robots based on a pre-set formation strategy or a real-time formation strategy formulated from the twin system, combined with the target information output by the visual positioning subsystem, and provides the navigator's motion constraint information to the navigator-follower formation module. The navigation-following formation module is used to determine the desired position and formation of each heterogeneous robot, and output it to the robot path planning subsystem; The robot control module is used to receive the desired path output by the robot path planning subsystem, decompose it into motion control commands for each of the heterogeneous robots, and send them to each of the heterogeneous robots for execution.

7. The hardware test platform for cooperative formation of multi-heterogeneous robot systems according to claim 6, characterized in that, The robot control module is also used to share the status information and control commands of each heterogeneous robot with the twin system.

8. The hardware test platform for collaborative formation of multi-heterogeneous robot systems according to claim 1, characterized in that, The preset path planning algorithm is the fast expanding random tree algorithm.

Images