A global information driven distributed multi-robot reinforcement learning formation and encirclement method based on 5G communication

Abstract

The application discloses a globally information-driven distributed multi-robot reinforcement learning formation and surrounding method based on 5G communication, which fully utilizes the characteristics of large bandwidth and low latency of 5G communication, collects sensing data of multiple unmanned aerial vehicles to a centralized edge server and reconstructs global state information through an algorithm, adopts a distributed strategy network in the motion decision stage to improve the scalability of the framework, so that the algorithm can adapt to a certain range of varying number of unmanned aerial vehicles without retraining, the scalability brought by the distributed kernel of the algorithm facilitates the deployment of the strategy trained on a small number of robots on a large number of robots, reduces the training cost of the reinforcement learning strategy, improves the adaptability of the hunting control algorithm, and finally improves the surrounding effect of the method and reduces the surrounding error.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-robot cooperative formation and encirclement, in particular to a global information driven distributed multi-robot reinforcement learning formation and encirclement method based on 5G communication. BACKGROUND

[0002] With the development of computer, complex system science, and robot technology, multi-robot system (MRS) is often used to complete various complex tasks that single robot system cannot or is difficult to complete due to its adaptability to the environment, high efficiency, low cost, and robustness in completing tasks. Multi-robot target encirclement task is a typical case, which is applied in important scenarios such as military attack and defense, post-disaster rescue, etc. For this problem, the classical solution mainly includes methods based on potential field function, artificial physics, behavior, and rules. These methods are highly dependent on artificial design and prior knowledge, and have poor adaptability and robustness in complex dynamic multi-interaction scenarios, so a multi-agent deep reinforcement learning method is considered to learn the encirclement strategy by interacting with the environment.

[0003] For multi-agent reinforcement learning methods, there are mainly two frameworks: centralized and distributed. The centralized framework means that multiple agents are modeled as an augmented single agent, the state space of which is the global information collected by the observations of multiple agents, the action space of which is the joint action of multiple agents, and there is only one centralized policy network that maps from the overall state space to the overall action space. The advantage is that it is easy to produce cooperative behavior with global information, which is beneficial to the maximization of the joint reward of the total task. However, the disadvantage is that the dimensions of the state and action space increase exponentially with the number of agents, which makes the policy training difficult and increases the computational burden. In theory, the requirement for strict synchronization of the observations of multiple agents also increases the communication burden in actual deployment. Meanwhile, unless a graph neural network or other module with variable input and output capability is introduced, the scalability of the centralized policy network to the number of agents is poor. The distributed framework means that each agent has an independent policy network that maps the local observation of a single agent to its independent action. The advantage is that the single policy network is easy to train due to the low dimensions of the state and action space, especially in scenarios where the agents and tasks are highly homogeneous, the training efficiency can be improved by sharing parameters. And the distributed framework is not sensitive to the number of agents and has strong scalability, allowing asynchronous information between agents, which reduces the actual communication burden. However, the disadvantage is that the distributed learning based on local observation is prone to local optimization and is not conducive to learning complex cooperative behavior.

[0004] Consider integrating the advantages of the above two frameworks to solve the problem of multi-robot formation encirclement: first, take advantage of the high bandwidth and low latency performance of the 5G communication system to efficiently aggregate the state observations of multiple robots (GPS, IMU, image and other sensor information) and target information (target position and speed information obtained by the observer) to a centralized edge server in real time. The global-local conversion module is used to obtain the neighbor and target state information in the local coordinate system of each machine. Then, a distributed multi-machine formation encirclement planner based on parameter sharing and parallel computing is used to generate speed planning instructions for each UAV. Finally, the control instructions are efficiently issued to each robot terminal through the 5G communication system, enabling the multi-mobile robot system to have the ability to implement collaborative formation encirclement of non-cooperative mobile targets. Summary of the Invention

[0005] The present invention addresses the problems existing in the prior art and provides a distributed multi-robot reinforcement learning formation encirclement method driven by global information based on 5G communication. It utilizes the performance advantages of 5G communication's high bandwidth and low latency to aggregate global information, and uses the information advantages of global information to drive a distributed multi-machine formation encirclement planner that considers neighbor information to generate speed planning instructions for each UAV. Finally, these instructions are transmitted to each robot terminal through 5G communication, enabling the multi-mobile robot system to have the ability to implement collaborative formation encirclement of non-cooperative mobile targets.

[0006] The present invention is achieved through the following technical solutions:

[0007] The present invention discloses a distributed multi-robot reinforcement learning formation encirclement method driven by global information based on 5G communication, comprising:

[0008] Receive multiple drone-borne sensor data (GPS, IMU, RGB images) sent by the multi-robot target encirclement system via the 5G communication system to the centralized edge server;

[0009] The sensor information collected by the centralized edge server is fed into the global information-driven distributed multi-robot reinforcement learning formation encirclement algorithm module running on it for processing, and the speed planning instructions for each drone are obtained;

[0010] The speed planning instructions for each drone are sent to the corresponding drone through the 5G communication system. Multiple drones form a disc-shaped formation to encircle non-cooperative moving targets, and follow the overall movement of the target to keep it at the center of the encirclement.

[0011] The global information-driven distributed multi-robot reinforcement learning formation encirclement algorithm module running on the centralized edge server processes the sensor information of the multi-robot target encirclement system as follows:

[0012] Firstly, the global-local conversion module is sent in, wherein the nearest neighbor and coordinate conversion calculation, target detection, target state observation module calculation are respectively calculated, fused and summarized to obtain the nearest neighbor relationship information between unmanned aerial vehicles and the global estimation information for the encircled target, and further converted into the nearest neighbor and target state information in the local coordinate system of each unmanned aerial vehicle;

[0013] The converted nearest neighbor and target state information in the local coordinate system of each unmanned aerial vehicle is sent into the multi-machine cooperative encirclement motion planner, the linear velocity of the unmanned aerial vehicle in the EN direction under the ENU coordinate system is planned through the encirclement strategy network based on reinforcement learning, the linear velocity in the U direction under the ENU coordinate system and the angular velocity along the Yaw axis are planned through the visual servo motion planner to keep the target always in the center of the field of view, and the speed planning instruction of each unmanned aerial vehicle is summarized.

[0014] As a further improvement, the nearest neighbor and coordinate conversion calculation is specifically: the collected global observation information is converted into local information in the coordinate system of each unmanned aerial vehicle considering the nearest neighbor, the global observation information includes the camera images, GPS positioning, linear velocity measurement information and IMU attitude information of all N unmanned aerial vehicles (N=10 in the embodiment), the local observation of each intelligent agent obtained by conversion includes the position and velocity information of the nearest neighbor unmanned aerial vehicle in the body coordinate system of each unmanned aerial vehicle, for each unmanned aerial vehicle, the k global nearest neighbor unmanned aerial vehicles can be solved according to the positioning information by using the KNN (K nearest neighbor) algorithm (k=2 in the embodiment), and the position information is converted into the body coordinate system and spliced into a feature vector, since multiple nearest neighbors are involved, the splicing order needs to be considered to ensure that the information representation for the distributed control architecture has permutation invariance, therefore, the nearest neighbor information is spliced in the order from the nearest to the far relative to the machine.

[0015] As a further improvement, the target state observation calculation of the application is specifically: an asynchronous monocular image fusion target state estimation algorithm is used to fuse and reconstruct the position and velocity information of the target to be encircled in the world coordinate system in real time according to the position information of the target to be encircled in the image coordinate system observed by each unmanned aerial vehicle and the position and velocity information of the corresponding unmanned aerial vehicle, and further converted into the local coordinate system of each unmanned aerial vehicle, thereby providing input for the subsequent multi-unmanned aerial vehicle target encirclement control task; the target state estimation algorithm uses an extended Kalman filter, combines a target uniform motion model assumption and a target detection algorithm to estimate the position and velocity of the target.

[0016] As a further improvement, the target state estimation algorithm of the application is specifically: if the state of the target is Where f t is the position of the target at time t, v t is the velocity of the target at time t, and the target state propagation equation is established as follows:

[0017]

[0018] where Δt is the time interval between two consecutive target detection results, n f and n v are the noises of the motion model with respect to position and velocity, respectively, the state propagation equation is abstracted as

[0019] When the state propagation is finished, the target state needs to be updated by the pixel coordinate results of the target detection algorithm. For the ith UAV, if it detects the target at time t, the pixel coordinate of the center point of the target is calculated as and the observation equation is as follows:

[0020]

[0021] where and are the attitude and position of the ith UAV at time t, the attitude is obtained by the inertial measurement unit (IMU) on the UAV, and the position can be obtained by the GPS, n uv is the observation noise, and Π is the re-projection equation of the camera, which converts the three-dimensional point coordinates in space into two-dimensional pixel coordinates in the image, the observation equation is abstracted as t y t = h(x uv ) + n t|t-1 ;

[0022] With the state propagation equation and the observation equation, the state estimation is performed by the standard extended Kalman filter, that is,

[0023]

[0024] P t = F t-1 F t T + Q t-1

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] where and P t-1are the state and covariance estimated at time t-1, and P t|t-1 are the predicted state and covariance at time t, Q t-1 is the noise covariance of , R t is the covariance of the observation noise, H t is the Jacobian matrix of the observation equation h with respect to the state .

[0031] As a further improvement, the encirclement strategy network in the multi-robot cooperative encirclement motion planner according to the present application is specifically:

[0032] The Actor network trained by the TD3-based reinforcement learning algorithm framework: TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm is an off-policy reinforcement learning framework (including Actor and Critic networks) of Actor-Critic type, wherein the Critic network is responsible for evaluating the value of the decision result of the Actor network according to the current state, and is used to guide the training of the Actor network during the training process; the Actor network is responsible for mapping the state to the decision action, and updates (deterministic policy gradient) by maximizing the cumulative expected return evaluated by the Critic;

[0033] Specifically, the reinforcement learning problem of multi-robot cooperative encirclement is modeled as follows:

[0034] 1) Joint state space: [f(x,y),v(v x ,v y )]*N, the position and velocity of N omni-directional motion robots in the ENU (East North Up) coordinate system;

[0035] 2) Single robot observation space The position and velocity of the target and k nearest neighbor robots (k=2 in this embodiment) relative to the body robot;

[0036] 3) Single robot action space: a t = [a x , a y ], which can be integrated to obtain a smooth velocity command to the underlying controller;

[0037] 4) Reward function design is the weighted sum of the following multiple rewards:

[0038] a. Attraction of the target annulus: r circle_attract = [||f c -f a||-radius] t-1 -[||f c -f a ||-radius] t

[0039] b. Repulsion between neighbor robots:

[0040] c. Total energy penalty: r circle =-||a t ||

[0041] d. Encirclement time reward: r on_circle =C on_circle

[0042] e. Collision penalty: r collision =-C collision

[0043] where f represents a two-dimensional position vector, v represents a two-dimensional linear velocity vector, a represents a two-dimensional linear acceleration vector, and radius is the enclosing radius;

[0044] The training phase: the method of sharing strategy parameters is used to share network parameters among multiple agents, reducing N strategy networks that need to be trained respectively to one, and multiple agents share the collected replay buffer experience pool during the exploration phase, improving the data sampling efficiency and experience richness of the exploration phase and accelerating the training process.

[0045] The reasoning phase: the centralized information is used but the distributed framework is adopted, that is, each agent is assigned an Actor strategy network trained by the TD3 reinforcement learning framework (parameter sharing strategy is used between networks), and all run at a frequency of 10HZ; the local observation feature vector calculated by the global-local conversion module from the current and past two frames is spliced as input (so that the strategy network has a prediction function), and is sent into each Actor strategy network to be mapped into [a x ,a y ](linear acceleration of the UAV in the ENU system), and after integration, the smooth speed command [v x ,v y ](linear velocity of the UAV in the ENU system) can be obtained.

[0046] The global information driven distributed multi-robot reinforcement learning formation and enclosing method based on 5G communication is realized by the following device:

[0047] 5G communication system: composed of 5G communication modules equipped by each terminal and a 5G private network formed by the modules;

[0048] Centralized edge server: composed of a computing platform equipped with high-performance CPU and GPU and a server-side 5G communication system;

[0049] Multi-robot target encirclement system: composed of N quadcopters equipped with 5G communication systems and equipped with visual perception and global positioning systems;

[0050] Target system to be encircled: composed of a red balloon marker and a carrier drone controlled by an operator to produce non-cooperative motion, which has non-cooperative motion behavior with the multi-robot system;

[0051] The scheme for the encirclement test based on the formation encirclement device: first, establish a 5G communication system connection between the centralized edge server and the multi-robot target encirclement system, and the target system to be encircled is controlled by the operator to fly to a certain height and make non-cooperative motion in a two-dimensional plane. After the N drones (N=10 in this embodiment) of the multi-robot target encirclement system take off at random positions nearby to a similar height, the entire system enters an automatic operation mode, that is, each drone transmits its sensing information to the centralized edge server through the 5G communication system, and the centralized edge server calculates the control instructions for each drone through the global information driven distributed multi-robot reinforcement learning formation encirclement method disclosed in the present application and issues the control instructions through the 5G communication system. This cycle continues until the final disc-shaped formation encirclement task of the target system to be encircled is completed according to the pre-set radius.

[0052] The beneficial effects of the present application are as follows:

[0053] The global information driven distributed framework proposed in the present application combines the advantages of centralized and distributed frameworks in multi-agent reinforcement learning and solves the problem of multi-robot formation encirclement by complementing and avoiding disadvantages: on the one hand, it fully utilizes the characteristics of large bandwidth and low latency of 5G communication to collect sensing data of multiple drones to the centralized edge server and reconstruct global state information through algorithms, and on the other hand, it adopts a distributed strategy network in the motion decision-making stage to improve the scalability of the framework, so that the algorithm can adapt to a certain range of varying number of drones without retraining (theoretically, the upper limit of the number of drones is only limited by the communication bandwidth and server computing power). Thanks to the scalability of the distributed kernel of the algorithm, the strategy trained on a small number of robots can be easily deployed on a large number of robots, reducing the training cost of the reinforcement learning strategy, improving the adaptability of the encirclement control algorithm, and ultimately improving the encirclement effect of the method, reducing the encirclement error, and the physical experiment conducted on the disclosed global information driven distributed multi-robot reinforcement learning formation encirclement device based on 5G communication has proved these beneficial effects. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1is a whole communication architecture diagram based on 5G;

[0055] Figure 2 is a global-local conversion module framework diagram;

[0056] Figure 3 is a target state estimation algorithm whole flow chart of multi-machine asynchronous monocular image fusion;

[0057] Figure 4 is a multi-machine cooperative encirclement motion planner framework diagram;

[0058] Figure 5 is the effect visualization diagram of the multi-robot system to the non-cooperative mobile target for disc-shaped formation encirclement physical experiment;

[0059] Figure 6 is the encirclement error of the multi-robot system to the non-cooperative mobile target for disc-shaped formation encirclement physical experiment. DETAILED DESCRIPTION

[0060] The technical solutions of the present application will be further described below by combining with the drawings in the specification and through specific embodiments:

[0061] The purpose of the present application is to solve the problems of poor scalability and high synchronization requirements when using a centralized framework and local information when using a distributed framework in multi-robot formation encirclement tasks, and to propose a global information driven distributed multi-robot reinforcement learning formation encirclement method based on 5G communication, which combines the advantages of the two frameworks to solve the multi-robot formation encirclement problem. The device for implementing the method of the present application includes a 5G communication system, a centralized edge server, a multi-robot target encirclement system, and a target to be encircled system, Figure 1 is a data flow diagram of the present application;

[0062] The specific implementation method of the present application is as follows:

[0063] Step one, first, each unmanned aerial vehicle transmits the on-board sensor data (GPS, IMU, RGB image) to the central 5G edge server through the 5G communication system Figure 1 ), and the specific implementation steps are as follows:

[0064] The on-board computer of each unmanned aerial vehicle obtains the local GPS, IMU and RGB image sensor data from the flight control and sensors through the MAVROS communication package, and then converts the subscription from the ROS topic to the most basic data type, such as int16 and float32, through the ROS-TCP forwarding node, and then packs the data according to the defined communication protocol format and adds a packet header (including data type, transmission priority and unmanned aerial vehicle ID), and uses the TCP transmission based on 5G communication to send to the edge server at a frequency of 10HZ.

[0065] The communication protocol is as follows:

[0066] {

[0067] 'Mtype': mtype,

[0068] 'Pri': pri,

[0069] 'Id': robot_id,

[0070] 'Dest': dest,

[0071] 'Data': data

[0072] }

[0073] The data is transmitted in a dictionary form. The 'Mtype' is the type of the message, and the user can define different data types, such as data sent by different sensors, for identification. The 'Pri' is the priority of the information, to determine the importance of the information. For example, it can send stop running and the like to all robots at the same time, and also coordinate the priority of forwarding between different data when the server resource is scarce. The 'Id' is the ID of all clients, each client has an independent ID, for distinguishing between each other. The 'Dest' is the destination of the client who wants to send the data, which is the ID of other registered clients, and can also use an empty address to let the cloud server automatically allocate. The above data are all the packet headers of the data packet. The 'Data' is the main content of the message, which is a binary byte stream.

[0074] The edge server end checks the IP and obtains the corresponding UAV ID by table lookup, and analyzes the data packet according to the communication protocol through the TCP-ROS forwarding node, and publishes it to the ROS of the edge server in the form of a ROS topic with frame_id and timestamp;

[0075] Step two, send the information collected on the 5G edge server into the global-local conversion module, fuse and summarize to obtain the nearest neighbor relationship information between UAVs and the global estimation information for the encircled target, and further convert it into the neighbor and target state information in the local coordinate system of each machine for subsequent distributed encirclement motion planner; The global-local conversion module is composed of nearest neighbor and coordinate conversion calculation, target detection, and target state observation modules: Figure 2

[0076] ​The nearest neighbor and coordinate conversion calculation is specifically: although it is a centralized framework at the information transmission level, in order to ensure the scalability of the subsequent control scheme, the collected global observation information needs to be converted into local information considering the neighbors in the coordinate system of each unmanned aerial vehicle. Specifically, the global observation includes the camera images, GPS positioning, line speed measurement information and IMU attitude information of all N unmanned aerial vehicles (N=10 in this embodiment), and the local observation of each intelligent agent obtained by conversion includes the position and speed information of the neighboring unmanned aerial vehicles in the body coordinate system of each unmanned aerial vehicle. For each unmanned aerial vehicle, its k global nearest neighbors can be solved by KNN (K nearest neighbor) algorithm according to the positioning information (n=2 in this embodiment), and the position information is converted to the local coordinate system to splice into a feature vector. Since multiple neighbors are involved, the splicing order needs to be considered to ensure that the information representation for the distributed control architecture has permutation invariance, so the neighbor information is spliced in the order from near to far relative to the local machine.

[0077] The target detection calculation is specifically: a target detection pre-training model based on YOLO V4 algorithm is used, and a detection model for the target to be enclosed is obtained by fine-tuning training on the target image data set to be enclosed. The model can detect the surrounding frame and center pixel coordinates of the target from the single frame RGB image returned by the unmanned aerial vehicle, which is used by the subsequent target state observation module and visual servo speed planner;

[0078] The target state observation calculation is specifically: an asynchronous monocular image fusion target state estimation algorithm ( Figure 3 ) is used to fuse and reconstruct the position and speed information of the target to be enclosed in the world coordinate system in real time according to the position information of the target to be enclosed observed by each unmanned aerial vehicle in the image coordinate system and the position, speed and attitude information of the corresponding unmanned aerial vehicle itself, and further convert it to the local coordinate system of each unmanned aerial vehicle, to provide input for the subsequent multi-unmanned aerial vehicle control task of multi-unmanned aerial vehicle target enclosure; since the process of transmitting images and self-sensing data by multiple unmanned aerial vehicles to the edge server is asynchronous (with time stamp), and due to the limited performance of the onboard computer, the frequency of information transmitted by a single machine to the server cannot be guaranteed. In view of the two problems of asynchronous transmission of multiple machines and non-uniform observation of a single machine, the target state estimation algorithm uses extended Kalman filter, combined with the assumption of uniform motion model of the target and the target detection algorithm to estimate the position and speed of the target. Specifically, the state of the target is assumed to be where f t is the position of the target at time t, and v t is the speed of the target at time t. The target state propagation equation is established as follows in this project:

[0079]

[0080] where Δt is the time interval between adjacent target detection results. nf and n v are the noises of the motion model with respect to position and velocity, respectively. The state propagation equation can be abstracted as

[0081] When the state propagation is finished, the target state needs to be updated with the pixel coordinate results of the target detection algorithm. Specifically, for the ith UAV, if it detects the target at time t, the pixel coordinate of the center point of the target can be calculated as Further, there is the following observation equation:

[0082]

[0083] where and are the attitude and position of the ith UAV at time t. The attitude can be obtained by the inertial measurement unit (IMU) on the UAV, and the position can be obtained by the GPS. n uv is the observation noise. Π is the re-projection equation of the camera, which converts the three-dimensional point coordinates in space to the two-dimensional pixel coordinates in the image. The observation equation can be abstracted as y t = h(x t ) + n uv .

[0084] With the state propagation equation and the observation equation, the state estimation can be performed by the standard extended Kalman filter, i.e.

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] where and P t-1 are the estimated state and covariance at time t-1, and P t|t-1 are the predicted state and covariance at time t, Q t-1 is the noise covariance of , R t is the covariance of the observation noise, and H t is the observation equation h with respect to the state the Jacobian matrix of the

[0093] After the above three parts of the calculation can be obtained in the local coordinate system of each UAV to be closed target position and speed and by the position of the nearest neighbor UAV position and speed information splicing into the feature vector for the third step based on reinforcement learning closing strategy network, the target detection calculation link to the target relative to the center of the field of view of each UAV pixel distance of the deviation of the third step of visual servo motion planner as input;

[0094] Step three, the conversion of the state information of the nearest neighbor and the target in the local coordinate system of each UAV is sent to the multi-machine cooperative closing motion planner Figure 4 ), through the closing strategy network based on reinforcement learning, the linear velocity of the UAV in the EN direction of the ENU coordinate system is planned, and the linear velocity and angular velocity in the U direction of the ENU coordinate system are planned through the visual servo motion planner to keep the target always in the center of the field of view, and the speed planning instructions of each UAV are obtained:

[0095] The closing strategy network based on reinforcement learning is specifically: the Actor network trained through the TD3-based reinforcement learning algorithm framework. TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm is an off-policy reinforcement learning framework of Actor-Critic type. The Critic network is responsible for evaluating the value of the decision result of the Actor network according to the current state, and is only used to guide the training of the Actor network in the training process, which is updated by minimizing the MSE error between the evaluation value and the target value; The Actor network is responsible for mapping the state to the decision action, and is updated by maximizing the cumulative expected return evaluated by the Critic (deterministic policy gradient), and all target networks in TD3 are updated in the form of soft update (EMA).

[0096] Specifically, the reinforcement learning problem of multi-machine cooperative closing is modeled as follows:

[0097] (1) Joint state space: [f(x,y),v(v x ,v y )]*N, the position and velocity of N omnidirectional mobile robots in the ENU (East North Up) coordinate system;

[0098] (2) Single robot observation space The position and velocity of the target and K nearest neighbor robots relative to the body robot;

[0099] (3) Single robot action space: a t= [a x ,a y ], after integration, the smooth speed command can be sent to the underlying controller;

[0100] (4) Reward function design is a weighted sum of the following multiple rewards:

[0101] a. Attraction of the target circle: r circle_attract = [||f c -f a ||-radius] t-1 -[||f c -f a ||-radiius] t

[0102] b. Repulsion between neighbor robots:

[0103] c. Total energy penalty: r circle = -||a t ||

[0104] d. Time reward in the circle: r on_circle =C on_circle

[0105] e. Collision penalty: r collision =-C collision

[0106] Where f represents a two-dimensional position vector, v represents a two-dimensional linear velocity vector, a represents a two-dimensional linear acceleration vector, and radius is the enclosing radius;

[0107] Training phase: Since the enclosing task is highly homogeneous for each UAV, the method of sharing network parameters among multiple agents is adopted, which is equivalent to reducing N strategy networks that need to be trained separately to one. During the exploration phase of training, multiple agents share the collected replay buffer experience pool, which greatly improves the data sampling efficiency and experience richness of the exploration phase and speeds up the training process. Thanks to the scalability of the distributed kernel, the strategy trained on a small number of robots can be used for large robot data processing, reducing the training cost of the reinforcement learning strategy and improving the adaptability of the hunting control algorithm;

[0108] Inference phase: Use centralized information but adopt a distributed framework, that is, assign each agent an Actor strategy network trained by the TD3 reinforcement learning framework (parameter sharing strategy is used between networks), and run at a frequency of 10HZ; Concatenate the local observation feature vectors calculated by step two into an input (so that the strategy network has a prediction function), and input it into each Actor strategy network to map to [a xa y ](linear acceleration of UAV in ENU system), and the smooth velocity command [v x ,v y ](linear velocity of UAV in ENU system) can be obtained by integration; thanks to the scalability brought by the distributed control framework, the strategy trained on a small number of robots can be directly applied to a large number of robots, reducing the training cost of the reinforcement learning strategy and improving the scalability and adaptability of the encirclement control algorithm;

[0109] The visual servo motion planner specifically maps the pixel distance of the target to be encircled relative to the center of the field of view of each UAV obtained in the target detection and calculation link in step two to the linear velocity of each UAV in the U direction and the angular velocity of rotation along the Yaw axis in the ENU coordinate system using a two-dimensional PID controller, so as to keep the target always in the center of the field of view of each UAV.

[0110] The output of the reinforcement learning-based encirclement strategy network and the visual servo motion planning is summarized to obtain the velocity planning command of each UAV.

[0111] Step four, run the calculation of steps two and three on the centralized edge server at a frequency of 10HZ, and finally obtain the velocity planning command through the 5G communication system. The command is executed by the corresponding UAV, and multiple UAVs implement disc-shaped formation encirclement on the non-cooperative moving target, and follow the target as a whole. The encirclement error converges quickly, and the target is always kept in the center of the encirclement (the encirclement effect is shown in Figure 5 ,6, Figure 5 The "X" in the figure represents the target to be encircled, and the ten solid black dots represent the ten UAVs in the encirclement system that are uniformly and accurately encircled on the disc with the target as the center and a radius of ten meters, Figure 6 The specific quantitative error is recorded, and the horizontal axis unit is 0.1s;

[0112] The application also discloses a global information driven distributed multi-robot reinforcement learning formation encirclement device based on 5G communication, comprising:

[0113] 5G communication system: composed of 5G communication modules equipped by each terminal and 5G private network formed by the modules, wherein the 5G communication module includes an industrial intelligent Internet of Things gateway (macro electric Z2 special 5G full-network industrial router) with 5G communication function, a 5G antenna and a China Telecom 5G Internet of Things card;

[0114] Centralized edge server: composed of a high-performance computing station equipped with a 12th Gen Core TM i7-12700K CPU and a NVIDIA GeForce RTX 3080Ti GPU and a server-side 5G communication system;

[0115] Multi-robot target encirclement system: composed of N self-made quadcopters (N=10 in this embodiment), a single quadcopter system is composed of a perception system, a communication system, a control system, and a power system; wherein the perception system of the quadcopter is mainly composed of an Intel RealSense D455 camera, a CUAV C-RTK positioning module, an IMU attitude measurement module, a magnetometer module, a barometer module, and a ground laser ranging module; the communication system of the quadcopter is a 5G communication system at the aircraft end; the control system of the quadcopter is mainly composed of a CUAV Pixhawk X7 Pro flight controller, a Raspberry Pi 4B onboard computer, and a siyi Siyi mk15 remote controller; the power system of the quadcopter is mainly composed of a battery, an electronic speed controller, a motor, and a power distribution and voltage distribution board;

[0116] Target to be encircled system: composed of a red balloon marker and a carrier drone with non-cooperative motion behavior controlled by an operator in addition to the multi-robot system; the marker is connected to the carrier drone by a rope, and after takeoff, the balloon can always automatically maintain 1 meter below the carrier drone under the downward airflow pressure generated by the rotor;

[0117] When performing the encirclement test, first, a 5G communication connection is established between the centralized edge server and the multi-robot target encirclement system, the target to be encircled system is controlled by the operator to take off to a certain height and make non-cooperative motion in a two-dimensional plane, and the N quadcopters (N=10 in this embodiment) of the multi-robot target encirclement system take off to a similar height at random positions nearby, and then the entire system enters an automatic operation mode, that is, each quadcopter transmits its sensing information to the centralized edge server through the 5G communication system, the centralized edge server calculates the control instructions for each quadcopter through the global information driven distributed multi-robot reinforcement learning formation and encirclement method disclosed in the present application deployed thereon, and the control instructions are issued through the 5G communication system, and the cycle continues until the circular formation encirclement task of the target to be encircled system according to the preset radius is completed.

[0118] Obviously, the above is not a limitation on the embodiments, and those skilled in the art can make various changes on the basis of the above description, and these changed embodiments are still within the protection scope of the present application.

Claims

1. A global information driven distributed multi-robot reinforcement learning formation encirclement method based on 5G communication, characterized in that, include: Receive GPS, IMU, and RGB image data from multiple drones sent by the multi-robot target encirclement system via the 5G communication system to the centralized edge server; The sensor information collected by the centralized edge server is fed into the distributed multi-robot reinforcement learning formation encirclement algorithm module running on it for processing, which then generates speed planning instructions for each drone. These speed planning instructions are then distributed to the corresponding drones via the 5G communication system for execution. Multiple drones then form a circular formation to encircle a non-cooperative moving target, following the target's overall movement and keeping it at the center of the encirclement. The global information-driven distributed multi-robot reinforcement learning formation encirclement algorithm module running on the centralized edge server processes the sensor information of the multi-robot target encirclement system as follows: First, it is sent to the global-local conversion module, where it is calculated and fused through the nearest neighbor and coordinate conversion calculation, target detection, and target state observation modules to obtain the nearest neighbor relationship information between drones and the global estimation information of the encircled target, which is further converted into the nearest neighbor and target state information in the local coordinate system of each drone. The converted neighbor and target state information in the local coordinate system of each UAV is fed into the multi-uav collaborative encirclement motion planner. The linear velocity of the UAV in the EN direction in the ENU coordinate system is planned by the reinforcement learning-based encirclement strategy network. The linear velocity in the U direction and the angular velocity along the Yaw axis in the ENU coordinate system are planned by the visual servoing motion planner to keep the target always in the center of the field of view. The speed planning instructions of each UAV are summarized.

2. The global information driven distributed multi-robot reinforcement learning formation encirclement method based on 5G communication according to claim 1, wherein, The nearest neighbor and coordinate conversion calculation is specifically as follows: the collected global observation information is converted into local information considering the nearest neighbors in the coordinate system of each drone. The global observation information includes all camera images, GPS positioning, linear velocity measurement information, and IMU attitude information of N drones. The converted local observations of each intelligent agent include the position and velocity information of the nearest neighbor drones in the coordinate system of each drone body. For each drone, its k global nearest neighbor drones can be solved using the KNN algorithm based on the positioning information, and their position information is converted to the coordinate system of the body and spliced ​​into a feature vector. The neighbor information is spliced ​​in the order from near to far relative to the drone body according to the need of permutation invariance.

3. The method of claim 1, wherein, The target state observation calculation is specifically as follows: a target state estimation algorithm based on asynchronous monocular image fusion is used to reconstruct the position and velocity information of the target to be encircled in the world coordinate system in real time based on the position information of the target to be encircled observed by each UAV in the image coordinate system and the position, velocity and attitude information of the corresponding UAV itself through a filtering algorithm, and further convert it to the local coordinate system of each UAV to provide input for the subsequent multi-UAV control task of encircling multiple UAV targets; The target state estimation algorithm uses extended Kalman filtering, combined with the target uniform motion model assumption and target detection algorithm to estimate the target position and velocity.

4. The method of claim 3, wherein, The target state estimation algorithm is specifically as follows: if the state of the target is Where f t is the position of the target at time t, v t is the speed of the target at time t, and a target state propagation equation is established as follows: where At is the time interval between two consecutive target detection results, n f and n v are the noise of the motion model with respect to position and velocity, respectively, and the state propagation equation is abstracted as When the state propagation is completed, the pixel coordinate result of the target detection algorithm is used to update the target state. For the ith UAV, if it detects the target at time t, the pixel coordinate of the center point of the target is calculated Further, there is an observation equation as follows: wherein and is the pose and position of the ith drone at time t, the pose is obtained by an inertial measurement unit (IMU) on the drone, the position is obtained by a GPS, n uv is an observation noise, Π is a re-projection equation of the camera, which converts three-dimensional point coordinates in space to two-dimensional pixel coordinates in the image, the observation equation is abstracted as y t = h(x t ) + n uv ; With the state propagation equation and observation equation, the state estimation is performed through the standard extended Kalman filter, that is, P t|t-1 = F t P t-1 F t T + Q t-1 S t =H t P t|t-1 H t T +R t P t = (I - K t H t )P t|t-1 where and P t-1 are the estimated state and covariance at time t-1, respectively, and P t|t-1 are the predicted state and covariance at time t, respectively, Q t-1 is the noise covariance of n Ft-1 , R t is the covariance of the observation noise, H t is the Jacobian matrix of the observation equation h with respect to the state .

5. The method of claim 1, wherein, The surrounding strategy network in the multi-machine cooperative surrounding motion planner is specifically: The Actor network trained by the TD3-based reinforcement learning algorithm framework: TD3 algorithm is an off-policy reinforcement learning framework, wherein the Actor network as the surrounding strategy network in the algorithm maps the input state to the output action decision, and the Critic network is responsible for evaluating the value of the Actor network decision result according to the input state in the training, guiding the training; The reinforcement learning problem modeling for multi-machine cooperative surrounding is as follows: 1) Joint state space is: [f(x,y), v(v x ,v y )]*N,N position and velocity of omnidirectional mobile robots in ENU (East North Up) coordinate system; 2) Single robot observation space Target and k-nearest neighbor robot position and velocity relative to body robot 3) Single robot action space: a t = [a x , a y ], after integration, smooth velocity commands are issued to the underlying controller; 4) The reward function design is a weighted sum of the following multiple rewards: a. Target ring attraction: r circle_attract = [||f c -f a ||-radius] t-1 = [||f c -f a ||-radius] t b. Repulsion between neighboring robots: c. Total energy penalty: r circle = -||a t || d. In-loop time reward: r on_circle = C on_circle e. Collision penalty: r collision = -C collision Wherein, f represents a two-dimensional position vector, v represents a two-dimensional linear velocity vector, a represents a two-dimensional linear acceleration vector, and radius is the surrounding radius; Training phase: The method of sharing strategy parameters is used to share network parameters among multiple agents, reducing N strategy networks that need to be trained respectively to one, and multiple agents share the collected replay buffer experience pool during the exploration stage of the training, improving the data sampling efficiency and experience richness of the exploration stage and accelerating the training process; Inference stage: Utilize centralized information but adopt distributed framework, assign each agent an actor policy network trained by TD3 reinforcement learning framework, all run at 10HZ frequency; Concatenate the local observation feature vector calculated by global-local conversion module from the current and past two frames as input to make the policy network have prediction function, input into each actor policy network to map to the linear acceleration of UAV in ENU system [a x , a y ] which can be integrated to get the smooth speed command linear velocity of UAV in ENU system [v x , v y ].

Images