Vision-based heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method

Abstract

The invention relates to a vision-based heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method, which is divided into two stages, in the first stage, a multi-rotor unmanned aerial vehicle without a visual sensor flies to a position which can be observed by the visual sensor of the multi-rotor unmanned aerial vehicle with the visual sensor, and in the second stage, the multi-rotor unmanned aerial vehicle without the visual sensor flies to a position which can be observed by the visual sensor of the multi-rotor unmanned aerial vehicle with the visual sensor. The multi-rotor unmanned aerial vehicle with the visual sensor guides the multi-rotor unmanned aerial vehicle without the visual sensor to fly to the interested target; the problem that part of unmanned aerial vehicles without visual sensors in an unmanned aerial vehicle cluster cannot execute a task based on visual servo control is solved, and by means of the method, part of unmanned aerial vehicles without visual sensors in a heterogeneous unmanned aerial vehicle cluster can also complete a task depending on the visual sensors of the unmanned aerial vehicles. By reducing the number of the unmanned aerial vehicles provided with the visual sensors, the use cost of the unmanned aerial vehicle cluster is reduced, meanwhile, the weight of the unmanned aerial vehicles is reduced, and the endurance time of the unmanned aerial vehicles is prolonged.

Description

Technical Field

[0001] The present invention relates to the field of multi-rotor unmanned aerial vehicle (UAV) cluster collaborative control, and in particular to a vision-based heterogeneous multi-rotor unmanned aerial vehicle (UAV) cluster collaborative guidance and control method. Background Art

[0002] In the invention patent "Visual servo target tracking control method for quad-rotor unmanned aerial vehicle based on preset performance", a visual servo target tracking control method for quad-rotor unmanned aerial vehicle with preset performance is proposed. The method is to obtain the spatial image coordinates and image plane coordinates of the target feature points through the airborne camera of the quad-rotor unmanned aerial vehicle, obtain the focal length of the airborne camera of the quad-rotor unmanned aerial vehicle, calculate the image features of the target feature points and the Jacobian matrix of the target feature points, form the image feature vector with the image features of multiple target feature points, form the Jacobian matrix of multiple target feature points into a superposition matrix, obtain the speed of the airborne camera, and convert the camera speed into the flight speed of the unmanned aerial vehicle through the coordinate system conversion, so as to realize the visual servo control of the unmanned aerial vehicle. At the same time, the method makes a performance preset in advance and makes certain constraints on the control system to avoid the control amount being too large, causing the target to leave the camera's field of view, resulting in the failure of the control method. The purpose of this method is to solve the problem that the target tracked by the unmanned aerial vehicle is easily lost due to the viewing angle limitation of the airborne camera and the coupling of the unmanned aerial vehicle movement with the airborne camera in the visual servo control of the quad-rotor unmanned aerial vehicle, thereby causing the mission to fail.

[0003] The above invention is effective only if the drone must be equipped with a camera. It is not applicable to drones that are not equipped with a camera. However, in a drone cluster, not every drone will be equipped with a camera. Installing a camera on every drone will inevitably increase the cost of use. At the same time, the weight of the camera will also reduce the drone's flight time. Therefore, the above invention is not the best method for drone clusters.

[0004] Visual sensors provide drones with rich environmental information such as color and shape, which enables drones to achieve functions such as target recognition and tracking. Therefore, visual sensors are crucial to enhancing the capabilities of drones. In mission scenarios such as search and rescue and reconnaissance, drone clusters can improve the efficiency of task execution compared to single drones. However, in a drone cluster, if each drone is equipped with a visual sensor, it will inevitably lead to waste of resources and increase in costs. In view of this, a common practice is to install visual sensors on only some drones and share visual data with the entire cluster through a communication network. However, this method has limitations, that is, drones that are not directly equipped with visual sensors may find it difficult to use existing visual servo control algorithms to complete specific tasks, such as target tracking, aperture crossing, etc. Summary of the invention

[0005] Aiming at the problem that some drones in a drone cluster that are not equipped with visual sensors cannot perform visual servo control tasks, a vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters is proposed. This method aims to support collaborative guidance and control of heterogeneous multi-rotor drone clusters (heterogeneous means that some drones in the cluster are equipped with visual sensors, while others are not), that is, drones in the cluster that are not equipped with visual sensors can also perform visual servo control-based tasks such as target tracking and aperture traversal.

[0006] The technical solution of the present invention is:

[0007] A vision-based collaborative guidance and control method for a cluster of heterogeneous multi-rotor drones is divided into two stages. In the first stage, a multi-rotor drone without a visual sensor flies to a position that can be observed by a visual sensor of a multi-rotor drone with a visual sensor. In the second stage, a multi-rotor drone with a visual sensor guides a multi-rotor drone without a visual sensor to fly to a target of interest. The method comprises the following steps:

[0008] Phase 1

[0009] Step 1: The drone cluster is powered on and started. Each drone in the drone cluster can obtain its position and attitude in the global coordinate system in real time. The multi-rotor drones equipped with visual sensors start working, and the multi-rotor drones without visual sensors wait for instructions sent by the multi-rotor drones equipped with visual sensors.

[0010] Step 2: The multi-rotor drone equipped with a visual sensor searches for the target of interest that needs to be tracked and crossed, uses the real-time image acquired by the visual sensor to check whether the target of interest exists through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system;

[0011] Step 3: After the multi-rotor drone equipped with a visual sensor finds the target of interest, it hovers and keeps its position and attitude unchanged. Based on the position and attitude in the current global coordinate system, it calculates the position of the point in front of the drone in the body coordinate system in the global coordinate system, and sends the position and current attitude angle to a multi-rotor drone in the cluster that is not equipped with a visual sensor through the communication middleware.

[0012] Step 4: After receiving the position and the attitude angle of the multi-rotor drone equipped with the visual sensor, the multi-rotor drone without the visual sensor flies to the position in the global coordinate system and adjusts its attitude during the flight to be consistent with the multi-rotor drone equipped with the visual sensor.

[0013] Phase II

[0014] Step 5: The visual sensor of the multi-rotor drone equipped with the visual sensor can simultaneously obtain the multi-rotor drone without the visual sensor and the target of interest, calculate their image deviations in the image coordinate system, convert the deviations into the body coordinate system, and then send them to the multi-rotor drone without the visual sensor through the communication network.

[0015] Step 6, the multi-rotor drone without a visual sensor is based on the received image deviation and the attitude angle of the multi-rotor drone with a visual sensor, and then the multi-rotor drone position and attitude controller are used to calculate the rotation speed of each motor, so that the attitude angle of the multi-rotor drone without a visual sensor is consistent with that of the multi-rotor drone with a visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster;

[0016] Step 7: Repeat steps 3 to 6 to achieve collaborative guidance and control of other multi-rotor drones in the cluster that are not equipped with visual sensors.

[0017] Further, the method specifically includes the following steps:

[0018] 1. Phase 1

[0019] Step 1: The drone cluster is powered on and started. Each drone in the drone cluster can obtain its position and attitude in the global coordinate system in real time. The multi-rotor drones equipped with visual sensors start working, and the multi-rotor drones without visual sensors wait for instructions sent by the multi-rotor drones equipped with visual sensors.

[0020] Step 2: The multi-rotor drone equipped with a visual sensor searches for the target of interest that needs to be tracked and crossed, uses the real-time image acquired by the visual sensor to check whether the target of interest exists through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system;

[0021] Step 3: After the multi-rotor drone equipped with a visual sensor finds the target of interest, it hovers and keeps its position and attitude unchanged based on the position and attitude in the current global coordinate system. Calculate the body coordinate system The position of the point in front of the drone in the global coordinate system Will The current attitude angle is sent to a multi-rotor drone in the cluster that is not equipped with a visual sensor through the communication middleware. The calculation is as follows:

[0022]

[0023] In formula (1), is the rolling angle φ of the multi-rotor drone equipped with a visual sensor in the global coordinate system t , pitch angle θ t and the yaw angle ψ t Construct the rotation matrix from the body coordinate system to the global coordinate system; is the translation matrix from the body coordinate system to the global coordinate system, x t ,y t 、z t They are the components of the x, y, and z axes of the multirotor drone equipped with a visual sensor in the global coordinate system; is a point in front of the multirotor drone equipped with a visual sensor, L x , L y , L z It is the component of the point on the x, y, and z axes in the body coordinate system;

[0024] Step 4: The multirotor drone without a visual sensor receives After measuring the attitude angle of the multi-rotor drone equipped with a visual sensor, it flies to the global coordinate system. This point, and adjust the attitude during flight to keep consistent with the multi-rotor drone equipped with visual sensors;

[0025] 2. Second Stage

[0026] Step 5: The visual sensor of the multi-rotor drone equipped with a visual sensor can simultaneously obtain the multi-rotor drone without a visual sensor and the target of interest, and calculate their image deviation in the image coordinate system. Then Convert to the body coordinate system Then it is sent to the multi-rotor drone without visual sensors through the communication network. and The calculation is as follows:

[0027]

[0028] In formula (2), (u t ,v t ) is the center position of the object of interest in the image coordinate system; (u q ,v q ) is the center position of the multi-rotor drone without a visual sensor in the image coordinate system; in formula (3), f is the focal length of the visual sensor, which is the internal parameter of the visual sensor and is a known constant;

[0029] Step 6: The multirotor drone without visual sensor is based on the received image deviation After calculating the attitude angle of the multi-rotor drone equipped with a visual sensor, the linear velocity in the global coordinate system and the angular velocity in the body coordinate system of the multi-rotor drone without a visual sensor are calculated by the PID controller. Then, the speed of each motor is calculated using the multi-rotor drone position and attitude controller, so that the attitude angle of the multi-rotor drone without a visual sensor is consistent with that of the multi-rotor drone equipped with a visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster.

[0030] The PID controller design of the linear speed in the body coordinate system is:

[0031]

[0032] In formula (4), and The proportional, integral and differential parameters of the PID controller for the linear speed need to be tuned;

[0033] The PID controller design of the angular velocity in the body coordinate system is:

[0034]

[0035] In formula (5), φ t ′ ,θ t ′ , t ′ They are the roll angle, pitch angle and heading angle without visual sensor installed; and The proportional, integral and differential parameters of the PID controller for angular velocity need to be tuned;

[0036] v b Through the global coordinate system φ t ′ ,θ t ′ , t ′ Constructed rotation matrix Transformed to the global coordinate system, the linear velocity calculation formula in the global coordinate system is:

[0037]

[0038] In formula (6), v w It is the speed control instruction of the multi-rotor UAV without visual sensor installed in the global coordinate system;

[0039] r b 、v w The multi-rotor drone position and attitude controller is provided to calculate the rotation speed of each motor by the position and attitude controller to realize the motion control of the drone, so that the attitude angle of the multi-rotor drone without visual sensor is consistent with that of the multi-rotor drone with visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster;

[0040] Step 7: Repeat steps 3 to 6 to achieve collaborative guidance and control of other multi-rotor drones in the cluster that are not equipped with visual sensors.

[0041] Furthermore, each drone in the cluster can calculate its own position and posture in the global coordinate system in real time through a motion capture system or multi-sensor fusion positioning technology. Each drone in the cluster sends its own position and posture to other drones in the cluster through the communication middleware, and obtains the position and posture of other drones in the global coordinate system in real time.

[0042] Furthermore, the required coordinate systems include a global coordinate system, a body coordinate system, and an image coordinate system, wherein the global coordinate system and the body coordinate system are right-handed systems, the image coordinate system represents the projection of a three-dimensional object on the image plane, the coordinate origin is the upper left corner of the image, the u-axis is parallel to the image plane and horizontally to the right, the v-axis is perpendicular to the u-axis and downward, the x-axis directions of the global coordinate system and the body coordinate system are consistent with the optical axis direction of the visual sensor, the origin of the global coordinate system is determined by the cluster positioning method, and the origin of the body coordinate system is the optical center position of the visual sensor.

[0043] Preferably, the position and attitude of the drone cluster in the global coordinate system are provided by a motion capture system, and the real-time center position of the balloon and the multi-rotor drone without a visual sensor installed in the image coordinate system is calculated using the yolov5 target recognition algorithm based on a deep neural network. The cluster drones communicate through wireless routing, and the multi-rotor drone position and attitude controller selects PX4.

[0044] The beneficial effects of the present invention are:

[0045] The present invention proposes a vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters, which enables some drones in a heterogeneous drone cluster that are not equipped with visual sensors to complete tasks that rely on their own visual sensors. By reducing the number of drones equipped with visual sensors, the cost of using the drone cluster is reduced, and at the same time, the weight of the drones is reduced and the flight time of the drones is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the collaborative guidance and control method of heterogeneous UAV clusters of the present invention;

[0047] Figure 2 This is a flow chart of the collaborative guidance and control method of heterogeneous UAV clusters of the present invention;

[0048] Figure 3 This is a verification example of the present invention - target search diagram;

[0049] Figure 4 This is an example of verification of the present invention - during the target search process, a drone equipped with a visual sensor recognizes a target of interest;

[0050] Figure 5 It is a verification example of the present invention - a collaborative guidance diagram;

[0051] Figure 6 This is an example of the verification of the present invention - a UAV equipped with a visual sensor identifies a target of interest during collaborative guidance and a UAV without a visual sensor installed;

[0052] Figure 7 This is a verification example of the present invention - a guide end diagram. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0054] The vision-based heterogeneous multi-rotor drone cluster collaborative guidance control method proposed in the present invention is divided into two stages during the execution process. In the first stage, the multi-rotor drone without a visual sensor flies to a position where the visual sensor of the multi-rotor drone with a visual sensor can observe. In the second stage, the multi-rotor drone with a visual sensor guides the multi-rotor drone without a visual sensor to fly to the target of interest. The whole process is as follows: Figure 2 shown.

[0055] In the present invention, there are a global coordinate system, a body coordinate system and an image coordinate system, wherein the global coordinate system and the body coordinate system are right-handed systems, the image coordinate system represents the projection of a three-dimensional space object on the image plane, the coordinate origin is the upper left corner of the image, the u-axis is parallel to the image plane and horizontally points to the right, the v-axis is perpendicular to the u-axis and points downward, the x-axis directions of the global coordinate system and the body coordinate system are consistent with the optical axis direction of the visual sensor, the origin of the global coordinate system is determined by the cluster positioning method, the origin of the body coordinate system is the optical center position of the visual sensor, and all coordinate systems are in Figure 1At the same time, in the present invention, each drone in the cluster can calculate its position and posture in the global coordinate system in real time through a motion capture system or a multi-sensor fusion positioning technology, and each drone in the cluster sends its position and posture to other drones in the cluster through a communication middleware, and obtains the position and posture of other drones in the global coordinate system in real time.

[0056] 1. Phase 1

[0057] Step 1: The drone cluster is powered on and started. Each drone in the drone cluster can obtain its position and attitude in the global coordinate system in real time. The multi-rotor drones equipped with visual sensors start working, and the multi-rotor drones without visual sensors wait for instructions sent by the multi-rotor drones equipped with visual sensors.

[0058] Step 2: A multirotor drone equipped with a visual sensor searches for the target of interest that needs to be tracked and traversed, uses the real-time image acquired by the visual sensor to check whether there is a target of interest through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system.

[0059] Step 3: After the multi-rotor drone equipped with a visual sensor finds the target of interest, it hovers and keeps its position and attitude unchanged based on the position and attitude in the current global coordinate system. Calculate the body coordinate system The position of the point in front of the drone in the global coordinate system Will The current attitude angle is sent to a multi-rotor drone in the cluster that is not equipped with a visual sensor through the communication middleware. The calculation is as follows:

[0060]

[0061] In formula (1), is the rolling angle φ of the multi-rotor drone equipped with a visual sensor in the global coordinate system t , pitch angle θ t and the yaw angle ψ t Construct the rotation matrix from the body coordinate system to the global coordinate system. is the translation matrix from the body coordinate system to the global coordinate system, x t ,y t 、z t They are the components of the x, y, and z axes of the multi-rotor drone equipped with visual sensors in the global coordinate system. is a point in front of the multirotor drone equipped with a visual sensor, L x , L y , L zIt is the component of the point on the x, y, and z axes in the body coordinate system.

[0062] Step 4: The multirotor drone without a visual sensor receives After measuring the attitude angle of the multi-rotor drone equipped with a visual sensor, it flies to the global coordinate system. This point, and adjust the posture during flight to keep consistent with the multi-rotor drone equipped with visual sensors.

[0063] 2. Second Stage

[0064] Step 5: The visual sensor of the multi-rotor drone equipped with a visual sensor can simultaneously obtain the multi-rotor drone without a visual sensor and the target of interest, and calculate their image deviation in the image coordinate system. Then Convert to the body coordinate system Then it is sent to the multi-rotor drone without visual sensors through the communication network. and The calculation is as follows:

[0065]

[0066] In formula (2), (u t ,v t ) is the center position of the object of interest in the image coordinate system. (u q ,v q ) is the center position of the multirotor drone without the visual sensor installed in the image coordinate system. In formula (3), f is the focal length of the visual sensor, which is the internal parameter of the visual sensor and a known constant.

[0067] Step 6: The multirotor drone without visual sensor is based on the received image deviation After obtaining the attitude angle of the multi-rotor UAV equipped with a visual sensor, the linear velocity in the global coordinate system and the angular velocity in the body coordinate system of the multi-rotor UAV without a visual sensor are calculated through the PID controller. Then, the speed of each motor is calculated using the existing mature multi-rotor UAV position and attitude controller (such as PX4, APM and other commercial flight control) to make the attitude angle of the multi-rotor UAV without a visual sensor consistent with that of the multi-rotor UAV equipped with a visual sensor, and move toward the center of the target of interest, thereby realizing the collaborative guidance and control of the heterogeneous cluster.

[0068] The PID controller design of the linear speed in the body coordinate system is:

[0069]

[0070] In formula (4), and The proportional, integral and differential parameters of the linear speed PID controller need to be tuned.

[0071] The PID controller design of the angular velocity in the body coordinate system is:

[0072]

[0073] In formula (5), φ t ′ ,θ t ′ , t ′ They are the roll angle, pitch angle and heading angle without visual sensor installed. and The proportional, integral and differential parameters of the angular velocity PID controller need to be tuned.

[0074] v b Through the global coordinate system φ t ′ ,θ t ′ , t ′ Constructed rotation matrix Transformed to the global coordinate system, the linear velocity calculation formula in the global coordinate system is:

[0075]

[0076] In formula (6), v w It is the speed control instruction of the multi-rotor UAV without visual sensor in the global coordinate system.

[0077] r b 、v w The multi-rotor UAV position and attitude controller is given to calculate the rotation speed of each motor by the position and attitude controller to realize the motion control of the UAV, so that the attitude angle of the multi-rotor UAV without visual sensor is consistent with that of the multi-rotor UAV with visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of heterogeneous clusters.

[0078] Step 7: Repeat steps 3 to 6 to achieve collaborative guidance and control of other multi-rotor drones in the cluster that are not equipped with visual sensors.

[0079] The method proposed in the present invention has been verified in real scenarios, which fully proves the feasibility of the heterogeneous cluster collaborative guidance control method proposed in the present invention. Figure 3 , 5, 7, in these figures, multi-rotor drones equipped with visual sensors, multi-rotor drones without visual sensors, and targets of interest are circled with ellipses and numbered. No. 1 is a multi-rotor drone equipped with a visual sensor, No. 2 is a multi-rotor drone without a visual sensor, and No. 3 is a target of interest. Figure 4 , 6 It is a real-time image obtained by a multi-rotor drone equipped with a visual sensor through its installed visual sensor. The multi-rotor drone without a visual sensor and the target of interest identified by the target recognition algorithm based on a deep neural network are marked in the image. The multi-rotor drone equipped with a visual sensor guides and controls the multi-rotor drone without a visual sensor to fly towards the target balloon of interest, and finally hits the balloon, causing the balloon to explode.

[0080] In this verification example, the position and attitude of the drone cluster in the global coordinate system are provided by the motion capture system. The target recognition algorithm based on the deep neural network of yolov5 is used to calculate the real-time center position of the balloon and the multi-rotor drone without visual sensors in the image coordinate system. The cluster drones communicate through wireless routing. The multi-rotor drone position and attitude controller selects PX4, L x =2.0 m, L y =0.0 m, L z =0.0 m, f = 3.05 mm,

[0081] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters, characterized in that: The method is divided into two stages. In the first stage, the multi-rotor drone without the visual sensor is flown to a position where the visual sensor of the multi-rotor drone with the visual sensor can observe. In the second stage, the multi-rotor drone with the visual sensor guides the multi-rotor drone without the visual sensor to fly to the target of interest. The method includes the following steps: Phase 1 Step 1: The drone cluster is powered on and started. Each drone in the drone cluster can obtain its position and attitude in the global coordinate system in real time. The multi-rotor drones equipped with visual sensors start working, and the multi-rotor drones without visual sensors wait for instructions sent by the multi-rotor drones equipped with visual sensors. Step 2: The multi-rotor drone equipped with a visual sensor searches for the target of interest that needs to be tracked and crossed, uses the real-time image acquired by the visual sensor to check whether the target of interest exists through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system; Step 3: After the multi-rotor drone equipped with a visual sensor finds the target of interest, it hovers and keeps its position and attitude unchanged. Based on the position and attitude in the current global coordinate system, it calculates the position of the point in front of the drone in the body coordinate system in the global coordinate system, and sends the position and current attitude angle to a multi-rotor drone in the cluster that is not equipped with a visual sensor through the communication middleware. Step 4: After receiving the position and the attitude angle of the multi-rotor drone equipped with the visual sensor, the multi-rotor drone without the visual sensor flies to the position in the global coordinate system and adjusts its attitude during the flight to be consistent with the multi-rotor drone equipped with the visual sensor. Phase II Step 5: The visual sensor of the multi-rotor drone equipped with the visual sensor can simultaneously obtain the multi-rotor drone without the visual sensor and the target of interest, calculate their image deviations in the image coordinate system, convert the deviations into the body coordinate system, and then send them to the multi-rotor drone without the visual sensor through the communication network. Step 6, the multi-rotor drone without a visual sensor is based on the received image deviation and the attitude angle of the multi-rotor drone with a visual sensor, and then the multi-rotor drone position and attitude controller are used to calculate the rotation speed of each motor, so that the attitude angle of the multi-rotor drone without a visual sensor is consistent with that of the multi-rotor drone with a visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster; Step 7: Repeat steps 3 to 6 to achieve collaborative guidance and control of other multi-rotor drones in the cluster that are not equipped with visual sensors.

2. The vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters according to claim 1 is characterized in that: The specific steps include:

1. Phase 1 Step 1: The drone cluster is powered on and started. Each drone in the drone cluster can obtain its position and attitude in the global coordinate system in real time. The multi-rotor drones equipped with visual sensors start working, and the multi-rotor drones without visual sensors wait for instructions sent by the multi-rotor drones equipped with visual sensors. Step 2: The multi-rotor drone equipped with a visual sensor searches for the target of interest that needs to be tracked and crossed, uses the real-time image acquired by the visual sensor to check whether the target of interest exists through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system; Step 3: After the multi-rotor drone equipped with a visual sensor finds the target of interest, it hovers and keeps its position and attitude unchanged based on the position and attitude in the current global coordinate system. Calculate the body coordinate system The position of the point in front of the drone in the global coordinate system Will The current attitude angle is sent to a multi-rotor drone in the cluster that is not equipped with a visual sensor through the communication middleware. The calculation is as follows: In formula (1), is the rolling angle Φ of the multi-rotor drone equipped with a visual sensor in the global coordinate system t , pitch angle θ t and the yaw angle ψ t Construct the rotation matrix from the body coordinate system to the global coordinate system; is the translation matrix from the body coordinate system to the global coordinate system, x t ,y t 、z t They are the components of the x, y, and z axes of the multirotor drone equipped with a visual sensor in the global coordinate system; is a point in front of the multirotor drone equipped with a visual sensor, L x , L y , L z It is the component of the point on the x, y, and z axes in the body coordinate system; Step 4: The multirotor drone without a visual sensor receives After measuring the attitude angle of the multi-rotor drone equipped with a visual sensor, it flies to the global coordinate system. This point, and adjust the attitude during flight to keep consistent with the multi-rotor drone equipped with visual sensors; 2. Second Stage Step 5: The visual sensor of the multi-rotor drone equipped with a visual sensor can simultaneously obtain the multi-rotor drone without a visual sensor and the target of interest, and calculate their image deviation in the image coordinate system. Then Convert to the body coordinate system Then it is sent to the multi-rotor drone without visual sensors through the communication network. and The calculation is as follows: In formula (2), (u t ,v t ) is the center position of the object of interest in the image coordinate system; (u q ,v q ) is the center position of the multi-rotor drone without a visual sensor in the image coordinate system; in formula (3), f is the focal length of the visual sensor, which is the internal parameter of the visual sensor and is a known constant; Step 6: The multirotor drone without visual sensor is based on the received image deviation After calculating the attitude angle of the multi-rotor drone equipped with a visual sensor, the linear velocity in the global coordinate system and the angular velocity in the body coordinate system of the multi-rotor drone without a visual sensor are calculated by the PID controller. Then, the speed of each motor is calculated using the multi-rotor drone position and attitude controller, so that the attitude angle of the multi-rotor drone without a visual sensor is consistent with that of the multi-rotor drone equipped with a visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster. The PID controller design of the linear speed in the body coordinate system is: In formula (4), and The proportional, integral and differential parameters of the PID controller for the linear speed need to be tuned; The PID controller design of the angular velocity in the body coordinate system is: In formula (5), φ t ′ ,θ t ′ , t ′ They are the roll angle, pitch angle and heading angle without visual sensor installed; and The proportional, integral and differential parameters of the PID controller for angular velocity need to be tuned; v b Through the global coordinate system φ t ′ ,θ t ′ , t ′ Constructed rotation matrix Transformed to the global coordinate system, the linear velocity calculation formula in the global coordinate system is: In formula (6), v w It is the speed control instruction of the multi-rotor UAV without visual sensor installed in the global coordinate system; r b 、v w The multi-rotor drone position and attitude controller is provided to calculate the rotation speed of each motor by the position and attitude controller to realize the motion control of the drone, so that the attitude angle of the multi-rotor drone without visual sensor is consistent with that of the multi-rotor drone with visual sensor, and moves toward the center of the target of interest, thereby realizing the collaborative guidance control of the heterogeneous cluster; Step 7: Repeat steps 3 to 6 to achieve collaborative guidance and control of other multi-rotor drones in the cluster that are not equipped with visual sensors.

3. The vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters according to claim 1 is characterized in that: Each drone in the cluster can calculate its own position and posture in the global coordinate system in real time through a motion capture system or multi-sensor fusion positioning technology. Each drone in the cluster sends its own position and posture to other drones in the cluster through the communication middleware, and obtains the position and posture of other drones in the global coordinate system in real time.

4. The vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters according to claim 1 is characterized in that: The required coordinate systems are the global coordinate system, the body coordinate system and the image coordinate system. The global coordinate system and the body coordinate system are right-handed systems. The image coordinate system represents the projection of three-dimensional objects on the image plane. The origin of the coordinate system is the upper left corner of the image. The u-axis is parallel to the image plane and horizontally to the right. The v-axis is perpendicular to the u-axis and downward. The x-axis direction of the global coordinate system and the body coordinate system is consistent with the optical axis direction of the visual sensor. The origin of the global coordinate system is determined by the cluster positioning method, and the origin of the body coordinate system is the optical center position of the visual sensor.

5. The vision-based collaborative guidance and control method for heterogeneous multi-rotor drone clusters according to claim 1 is characterized in that: The position and attitude of the drone cluster in the global coordinate system are provided by the motion capture system. The yolov5 target recognition algorithm based on deep neural network is used to calculate the real-time center position of the balloon and the multi-rotor drone without visual sensor in the image coordinate system. The cluster drones communicate through wireless routing, and PX4 is selected as the multi-rotor drone position and attitude controller.

Images