A visual-based cooperative guidance control method for heterogeneous multi-copter unmanned aerial vehicle cluster

Abstract

The present application relates to a vision-based heterogeneous multi-copter unmanned aerial vehicle cluster cooperative guidance control method, which is divided into two stages. In the first stage, the multi-copter unmanned aerial vehicle without a vision sensor flies to a position that can be observed by the vision sensor of the multi-copter unmanned aerial vehicle with a vision sensor. In the second stage, the multi-copter unmanned aerial vehicle with a vision sensor guides the multi-copter unmanned aerial vehicle without a vision sensor to fly to a target of interest. The method solves the problem that part of the unmanned aerial vehicles without a vision sensor in the unmanned aerial vehicle cluster cannot perform a vision-based servo control task. The method enables part of the unmanned aerial vehicles without a vision sensor in the heterogeneous unmanned aerial vehicle cluster to also complete a task that relies on the vision sensor itself. By reducing the number of unmanned aerial vehicles with a vision sensor, the use cost of the unmanned aerial vehicle cluster is reduced, and at the same time, the weight of the unmanned aerial vehicle is reduced, and the endurance time of the unmanned aerial vehicle is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-rotor unmanned aerial vehicle cluster cooperative control, and particularly relates to a heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method based on vision. BACKGROUND

[0002] In the invention patent "target tracking control method for four-rotor unmanned aerial vehicle based on preset performance visual servo", a target tracking control method for four-rotor unmanned aerial vehicle based on preset performance visual servo is proposed. The method is to obtain the spatial image coordinates of the target feature points and the image plane coordinates of the target feature points through the on-board camera of the four-rotor unmanned aerial vehicle, obtain the focal length of the on-board camera of the four-rotor unmanned aerial vehicle, calculate the image features of the target feature points and the Jacobian matrix of the target feature points, form the image feature vectors of the multiple target feature points, form the superposition matrix of the multiple target feature points, obtain the speed of the on-board camera, and convert the camera speed into the flight speed of the unmanned aerial vehicle through coordinate system conversion, so as to realize the visual servo control of the unmanned aerial vehicle. At the same time, the method makes performance preset in advance and makes certain constraints on the control system to avoid the control quantity being too large to cause the target to leave the field of view of the camera and cause the control method to fail. The purpose of the method is to solve the problem that in the visual servo control of the four-rotor unmanned aerial vehicle, the target tracked by the unmanned aerial vehicle is easily lost due to the limitation of the viewing angle of the on-board camera and the coupling of the motion of the unmanned aerial vehicle and the motion of the on-board camera, thereby causing the task to fail.

[0003] The above invention is effective on the premise that the unmanned aerial vehicle must be equipped with a camera, and is not applicable to unmanned aerial vehicles without a camera. However, in an unmanned aerial vehicle cluster, not every unmanned aerial vehicle is equipped with a camera, and equipping every unmanned aerial vehicle with a camera will inevitably increase the use cost, and the weight of the camera will also reduce the endurance time of the unmanned aerial vehicle. Therefore, the above invention is not the optimal method for the unmanned aerial vehicle cluster.

[0004] A vision sensor provides rich environmental information such as color and shape for an unmanned aerial vehicle, which enables the unmanned aerial vehicle to realize target recognition and tracking functions. Therefore, the vision sensor is crucial for enhancing the capability of the unmanned aerial vehicle. In search and rescue, reconnaissance and other task scenarios, compared with a single unmanned aerial vehicle, an unmanned aerial vehicle cluster can improve the execution efficiency of the task. However, in the unmanned aerial vehicle cluster, if every unmanned aerial vehicle is equipped with a vision sensor, it will inevitably lead to waste of resources and increase in cost. In view of this, a common approach is to equip only part of the unmanned aerial vehicles with vision sensors and share the vision data through a communication network to the entire cluster. However, this method has limitations, that is, those unmanned aerial vehicles without directly installed vision sensors may have difficulty in completing specific tasks such as target tracking, aperture crossing, etc. using existing visual servo control algorithms. SUMMARY

[0005] Aiming at the problem that part of unmanned aerial vehicles in unmanned aerial vehicle cluster cannot execute visual servo control task without installing visual sensor, a heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method based on vision is proposed. The method aims to support the cooperative guidance control of heterogeneous multi-rotor unmanned aerial vehicle cluster (heterogeneous refers to part of unmanned aerial vehicles in the cluster are installed with visual sensor and others are not installed), that is, to enable unmanned aerial vehicles without installing visual sensor to also execute target tracking and aperture traversal and other visual servo control tasks.

[0006] The technical scheme of the present application is:

[0007] A heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method based on vision, which is divided into two stages. In the first stage, the multi-rotor unmanned aerial vehicle without installing visual sensor flies to a position that can be observed by the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor. In the second stage, the multi-rotor unmanned aerial vehicle with visual sensor guides the multi-rotor unmanned aerial vehicle without visual sensor to fly to the target of interest; comprising the following steps:

[0008] First stage

[0009] Step 1, power on the unmanned aerial vehicle cluster, each unmanned aerial vehicle in the unmanned aerial vehicle cluster can obtain the position and attitude in the global coordinate system in real time, the multi-rotor unmanned aerial vehicle with visual sensor starts to work, and the multi-rotor unmanned aerial vehicle without visual sensor waits for the command sent by the multi-rotor unmanned aerial vehicle with visual sensor;

[0010] Step 2, the multi-rotor unmanned aerial vehicle with visual sensor searches for the target of interest that needs to be tracked and traversed, checks whether the target of interest exists by using the real-time image obtained by the visual sensor through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system;

[0011] Step 3, after the multi-rotor unmanned aerial vehicle with visual sensor discovers the target of interest, it hovers, keeps the position and attitude unchanged, calculates the position of the point in front of the unmanned aerial vehicle in the global coordinate system based on the current position and attitude in the global coordinate system, and sends the position and the current attitude angle to a multi-rotor unmanned aerial vehicle without visual sensor in the cluster through the communication middleware,

[0012] Step 4, after receiving the position and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor flies to the position in the global coordinate system and adjusts the attitude during the flight to keep consistent with the multi-rotor unmanned aerial vehicle with visual sensor;

[0013] Second stage

[0014] Step 5, the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor can simultaneously acquire the multi-rotor unmanned aerial vehicle without visual sensor and the target of interest, calculate the image deviation of them in the image coordinate system, convert the deviation to the body coordinate system, and then send it to the multi-rotor unmanned aerial vehicle without visual sensor through the communication network,

[0015] Step 6, based on the received image deviation and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor calculates the linear velocity in the global coordinate system and the angular velocity in the body coordinate system through the PID controller respectively, and then calculates the speed of each motor by using the multi-rotor unmanned aerial vehicle position and attitude controller, so that the attitude angle of the multi-rotor unmanned aerial vehicle without visual sensor is consistent with that of the multi-rotor unmanned aerial vehicle with visual sensor, and moves to the center of the target of interest, thereby realizing the cooperative guidance control of the heterogeneous cluster.

[0016] Step 7, repeat the above steps 3 to 6 to realize the cooperative guidance control of other multi-rotor unmanned aerial vehicles without visual sensor in the cluster.

[0017] Further, the specific steps include:

[0018] I. First stage

[0019] Step 1, the unmanned aerial vehicle cluster is powered on, each unmanned aerial vehicle in the unmanned aerial vehicle cluster can acquire the position and attitude in the global coordinate system in real time, the multi-rotor unmanned aerial vehicle with visual sensor starts to work, and the multi-rotor unmanned aerial vehicle without visual sensor waits for the instruction sent by the multi-rotor unmanned aerial vehicle with visual sensor;

[0020] Step 2, the multi-rotor unmanned aerial vehicle with visual sensor searches the target of interest which needs to be tracked and crossed, checks whether the target of interest exists by using the real-time image acquired by the visual sensor through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system;

[0021] Step 3, after the multi-rotor unmanned aerial vehicle with visual sensor finds the target of interest, it hovers, keeps the position and attitude unchanged, and calculates the position and attitude in the global coordinate system the position of the point in front of this unmanned aerial vehicle in the global coordinate system and the current attitude angle are sent to a multi-rotor unmanned aerial vehicle without visual sensor in the cluster through the communication middleware, The calculation method is as follows:

[0022] ​​

[0023] In formula (1), is the roll angle of the multi-rotor unmanned aerial vehicle with visual sensor in the global coordinate system t , the pitch angle t and the yaw angle t The rotation matrix of the body coordinate system to the global coordinate system is constructed; is the translation matrix of the body coordinate system to the global coordinate system, x t , y t , z t are the components of the multi-rotor unmanned aerial vehicle with visual sensor in the global coordinate system on the x, y, z three coordinate axes respectively; is a point in front of the multi-rotor unmanned aerial vehicle with visual sensor, L x , L y , L z are the components of the point in the body coordinate system on the x, y, z three coordinate axes;

[0024] Step 4, after receiving and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor flies to this point in the global coordinate system and adjusts the attitude during the flight to keep consistent with the multi-rotor unmanned aerial vehicle with visual sensor;

[0025] II. Second stage

[0026] Step 5, the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor can simultaneously obtain the multi-rotor unmanned aerial vehicle without visual sensor and the target of interest, calculate their image deviation in the image coordinate system Convert to the body coordinate system Then send it to the multi-rotor unmanned aerial vehicle without visual sensor through the communication network, and The calculation method is as follows:

[0027]

[0028] In formula (2), (u t , v t ) is the center position of the target of interest in the image coordinate system; (u q , v q ) is the center position of the multi-rotor unmanned aerial vehicle without visual sensor in the image coordinate system; In formula (3), f is the focal length of the visual sensor, and is the intrinsic parameter of the visual sensor, which is a known constant;

[0029] Step 6: The multi-rotor drone without a visual sensor, based on the received image deviation... After determining the attitude angles of the multi-rotor drones equipped with vision sensors, the linear velocity in the global coordinate system and the angular velocity in the body coordinate system of the multi-rotor drones without vision sensors are calculated by the PID controller. Then, the rotational speed of each motor is calculated using the position and attitude controller of the multi-rotor drones, so that the attitude angles of the multi-rotor drones without vision sensors are consistent with those of the multi-rotor drones with vision sensors, and they move towards the center of the target of interest, thereby realizing the cooperative guidance control of heterogeneous clusters.

[0030] The PID controller for linear velocity in the body coordinate system is designed as follows:

[0031]

[0032] In equation (4), and The proportional, integral, and derivative parameters of the PID controller for linear velocity need to be tuned.

[0033] The PID controller for angular velocity in the body coordinate system is designed as follows:

[0034]

[0035] In equation (5), φ t ′ θ t ′ ψ t ′ These are the roll angle, pitch angle, and yaw angle without visual sensors installed. and The proportional, integral, and derivative parameters of the PID controller for angular velocity need to be tuned.

[0036] v b φ in the global coordinate system t ′ θ t ′ ψ t ′ Constructed rotation matrix Transforming to the global coordinate system, the formula for calculating linear velocity in the global coordinate system is:

[0037]

[0038] In equation (6), v w Speed ​​control commands for multi-rotor drones without visual sensors in the global coordinate system;

[0039] r b v w The position and attitude controller of the multi-rotor unmanned aerial vehicle is given, the rotation speed of each motor is calculated by the position and attitude controller, the motion control of the unmanned aerial vehicle is realized, the attitude angle of the multi-rotor unmanned aerial vehicle without installing a visual sensor is consistent with the multi-rotor unmanned aerial vehicle with installing a visual sensor, and the center of the target of interest is moved, so that the cooperative guidance control of the heterogeneous cluster is realized.

[0040] Step 7, the above steps 3 to 6 are repeated to realize the cooperative guidance control of other multi-rotor unmanned aerial vehicles without installing a visual sensor in the cluster.

[0041] Further, each unmanned aerial vehicle in the cluster can calculate the position and attitude in the global coordinate system in real time through a motion capture system or a multi-sensor fusion positioning technology, and each unmanned aerial vehicle in the cluster sends the position and attitude to other unmanned aerial vehicles in the cluster through a communication middleware, and obtains the position and attitude of other unmanned aerial vehicles in the global coordinate system in real time.

[0042] Further, the required coordinate systems include a global coordinate system, a body coordinate system and an image coordinate system, wherein the global coordinate system and the body coordinate system are right-handed systems, the image coordinate system represents the projection of a three-dimensional space object on an image plane, the origin is the upper left corner of the image, the u-axis is parallel to the horizontal right of the image plane, and the v-axis is perpendicular to the u-axis downward, the x-axis direction of the global coordinate system and the body coordinate system is consistent with the optical axis direction of the visual sensor, and the origin of the global coordinate system is determined by the cluster positioning mode, and the origin of the body coordinate system is the optical center position of the visual sensor.

[0043] Preferably, the position and attitude of the unmanned aerial vehicle cluster in the global coordinate system are provided by a motion capture system, the real-time center position of the balloon and the multi-rotor unmanned aerial vehicle without installing a visual sensor in the image coordinate system is calculated by using a yolov5 target recognition algorithm based on a deep neural network, the unmanned aerial vehicles in the cluster communicate through a wireless router, and the position and attitude controller of the multi-rotor unmanned aerial vehicle selects PX4.

[0044] The present application has the advantages of:

[0045] The present application provides a heterogeneous multi-rotor unmanned aerial vehicle cluster cooperative guidance control method based on vision, which enables part of the unmanned aerial vehicles without installing a visual sensor in the heterogeneous unmanned aerial vehicle cluster to complete the task relying on the visual sensor itself. By reducing the number of unmanned aerial vehicles with visual sensors, the use cost of the unmanned aerial vehicle cluster is reduced, and at the same time, the weight of the unmanned aerial vehicle is reduced, and the endurance time of the unmanned aerial vehicle is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 It is a schematic diagram of the heterogeneous unmanned aerial vehicle cluster cooperative guidance control method.

[0047] Figure 2 Flow chart of the heterogeneous UAV cluster cooperative guidance control method of the present application;

[0048] Figure 3 Verification example-target search diagram of the present application;

[0049] Figure 4 Diagram of the UAV with visual sensor recognizing the target of interest in the verification example-target search process of the present application;

[0050] Figure 5 Verification example-cooperative guidance diagram of the present application;

[0051] Figure 6 Diagram of the UAV with visual sensor recognizing the target of interest and the UAV without visual sensor in the verification example-cooperative guidance process of the present application;

[0052] Figure 7 Verification example-end of guidance diagram of the present application. DETAILED DESCRIPTION

[0053] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments are implemented on the premise of the technical solution of the present application, and detailed implementation and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0054] The heterogeneous multi-rotor UAV cluster cooperative guidance control method based on vision proposed in the present application is divided into two stages in the execution process. In the first stage, the multi-rotor UAV without visual sensor flies to a position that can be observed by the visual sensor of the multi-rotor UAV with visual sensor. In the second stage, the multi-rotor UAV with visual sensor guides the multi-rotor UAV without visual sensor to fly to the target of interest. The whole process is as shown in Figure 2 .

[0055] In the present application, there are global coordinate system, body coordinate system and image coordinate system. The global coordinate system and the body coordinate system are right-handed systems, the image coordinate system represents the projection of a three-dimensional space object on the image plane, the coordinate origin is the upper left corner of the image, the u-axis is parallel to the horizontal right of the image plane, and the v-axis is perpendicular to the u-axis downward. The x-axis direction of the global coordinate system and the body coordinate system is consistent with the optical axis direction of the visual sensor. The global coordinate system origin is determined by the cluster positioning method, and the body coordinate system origin is the optical center position of the visual sensor. All coordinate systems are in Figure 1The winning mark. Meanwhile, in the present application, each unmanned aerial vehicle in the cluster can calculate its position and attitude in the global coordinate system in real time through the motion capture system or multi-sensor fusion positioning technology, and each unmanned aerial vehicle in the cluster sends its position and attitude to other unmanned aerial vehicles in the cluster through the communication middleware, and real-time acquires the position and attitude of other unmanned aerial vehicles in the global coordinate system.

[0056] I. First stage

[0057] Step 1, the unmanned aerial vehicle cluster is powered on, each unmanned aerial vehicle in the unmanned aerial vehicle cluster can acquire the position and attitude in the global coordinate system in real time, the multi-rotor unmanned aerial vehicle with visual sensor starts to work, and the multi-rotor unmanned aerial vehicle without visual sensor waits for the instruction sent by the multi-rotor unmanned aerial vehicle with visual sensor.

[0058] Step 2, the multi-rotor unmanned aerial vehicle with visual sensor searches the target of interest which needs to be tracked and crossed, checks whether the target of interest exists through the target recognition algorithm of the deep neural network using the real-time image obtained by the visual sensor, and calculates the center position of the target in the image system.

[0059] Step 3, after the multi-rotor unmanned aerial vehicle with visual sensor discovers the target of interest, it hovers, keeps the position and attitude unchanged, and calculates the position and attitude in the global coordinate system based on the current position and attitude in the global coordinate system The position of the point in front of the unmanned aerial vehicle in the global coordinate system The position of the point in front of the unmanned aerial vehicle in the global coordinate system The position of the point in front of the unmanned aerial vehicle in the global coordinate system And the current attitude angle are sent to a multi-rotor unmanned aerial vehicle without visual sensor in the cluster through the communication middleware, The calculation method is as follows:

[0060]

[0061] In formula (1), The roll angle φ of the multi-rotor unmanned aerial vehicle with visual sensor in the global coordinate system t The pitch angle θ t And the yaw angle ψ t The rotation matrix of the body coordinate system to the global coordinate system. The translation matrix of the body coordinate system to the global coordinate system, x t , y t , z t Are the components of the multi-rotor unmanned aerial vehicle with visual sensor in the global coordinate system. Is a point in front of the multi-rotor unmanned aerial vehicle with visual sensor, L x , L y , L zis the component of the point in the x, y, z coordinate axes in the body coordinate system.

[0062] Step 4, the multi-rotor unmanned aerial vehicle without visual sensor receives the image deviation and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, and flies to the point in the global coordinate system, and adjusts the attitude during the flight to keep consistent with the multi-rotor unmanned aerial vehicle with visual sensor.

[0063] II. Second stage

[0064] Step 5, the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor can simultaneously obtain the multi-rotor unmanned aerial vehicle without visual sensor and the target of interest, and calculate the image deviation of them in the image coordinate system and then convert to the body coordinate system and then send to the multi-rotor unmanned aerial vehicle without visual sensor through the communication network, and The calculation method is as follows:

[0065]

[0066] In formula (2), (u t ,v t ) is the center position of the target of interest in the image coordinate system. (u q ,v q ) is the center position of the multi-rotor unmanned aerial vehicle without visual sensor in the image coordinate system. In formula (3), f is the focal length of the visual sensor, and is the intrinsic parameter of the visual sensor, which is a known constant.

[0067] Step 6, based on the received image deviation and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor calculates the linear velocity in the global coordinate system and the angular velocity in the body coordinate system through the PID controller respectively, and then uses the existing mature multi-rotor unmanned aerial vehicle position and attitude controller (such as PX4, APM, etc. Commercial flight control) to calculate the speed of each motor, so that the attitude angle of the multi-rotor unmanned aerial vehicle without visual sensor is consistent with that of the multi-rotor unmanned aerial vehicle with visual sensor, and moves towards the center of the target of interest, thereby realizing the cooperative guidance control of the heterogeneous cluster.

[0068] The PID controller design of the linear velocity in the body coordinate system is:

[0069]

[0070] In formula (4), and PID controller of linear velocity, the proportional, integral and differential parameters need to be parameterized.

[0071] The PID controller of angular velocity under the body coordinate system is designed as:

[0072]

[0073] In formula (5), φ t ′ , theta t ′ , psi t ′ Respectively, the roll angle, pitch angle and heading angle of the visual sensor are not installed. and PID controller of angular velocity, the proportional, integral and differential parameters need to be parameterized.

[0074] v b Through the rotation matrix of phi t ′ , theta t ′ , psi t ′ The rotation matrix Transformed into the global coordinate system, the linear velocity calculation formula in the global coordinate system is:

[0075]

[0076] In formula (6), v w The speed control command of the multi-rotor unmanned aerial vehicle without installing a visual sensor in the global coordinate system.

[0077] R b , v w To the position and attitude controller of the multi-rotor unmanned aerial vehicle, the speed of each motor is calculated by the position and attitude controller, the motion control of the unmanned aerial vehicle is realized, the attitude angle of the multi-rotor unmanned aerial vehicle without installing a visual sensor is consistent with the multi-rotor unmanned aerial vehicle with installing a visual sensor, and the center of the target of interest is moved, so that the cooperative guidance control of the heterogeneous cluster is realized.

[0078] Step 7, repeat the above steps 3 to 6, realize the cooperative guidance control of other multi-rotor unmanned aerial vehicles without installing a visual sensor in the cluster.

[0079] The method proposed in the application is verified in a real scene, which fully proves the feasibility of the heterogeneous cluster cooperative guidance control method proposed in the application, such as Figure 3 , 5, 7 are shown, in which the multi-rotor unmanned aerial vehicle equipped with a visual sensor, the multi-rotor unmanned aerial vehicle without a visual sensor and the target of interest are circled with an ellipse and numbered, numbered 1 is the multi-rotor unmanned aerial vehicle equipped with a visual sensor, numbered 2 is the multi-rotor unmanned aerial vehicle without a visual sensor, and numbered 3 is the target of interest, Figure 4 、 6 is a real-time image obtained by the multi-rotor unmanned aerial vehicle equipped with a visual sensor through the visual sensor mounted thereon, in which the multi-rotor unmanned aerial vehicle without a visual sensor and the target of interest are marked, which are identified by the target recognition algorithm based on a deep neural network. The multi-rotor unmanned aerial vehicle equipped with a visual sensor guides the control of the multi-rotor unmanned aerial vehicle without a visual sensor to fly to the target balloon, and finally collides with the balloon, causing the balloon to explode.

[0080] In this verification example, the position and attitude of the unmanned aerial vehicle cluster in the global coordinate system are provided by the motion capture system, the real-time center position of the balloon and the multi-rotor unmanned aerial vehicle without a visual sensor in the image coordinate system is calculated by using the yolov5 target recognition algorithm based on a deep neural network, the cluster unmanned aerial vehicles communicate through wireless routing, and the multi-rotor unmanned aerial vehicle position and attitude controller selects PX4, x = 2.0 meters, L y = 0.0 meters, L z = 0.0 meters, f = 3.05 mm,

[0081] The above-described embodiments only express one embodiment of the present application, which is described in detail and in detail, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A vision-based cooperative guidance control method for a heterogeneous multi-copter UAV swarm, characterized in that, The method is divided into two stages, in the first stage, the multi-rotor unmanned aerial vehicle without visual sensor flies to a position which can be observed by the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor, in the second stage, the multi-rotor unmanned aerial vehicle with visual sensor guides the multi-rotor unmanned aerial vehicle without visual sensor to fly to the target of interest; comprising the following steps: The first stage Step 1, the unmanned aerial vehicle cluster is powered on, each unmanned aerial vehicle in the unmanned aerial vehicle cluster can obtain the position and attitude in the global coordinate system in real time, the multi-rotor unmanned aerial vehicle with visual sensor starts to work, and the multi-rotor unmanned aerial vehicle without visual sensor waits for the instruction sent by the multi-rotor unmanned aerial vehicle with visual sensor; Step 2, the multi-rotor unmanned aerial vehicle with visual sensor searches the target of interest which needs to be tracked and crossed, checks whether the target of interest exists by using the target recognition algorithm of the deep neural network based on the real-time image obtained by the visual sensor, and calculates the central position of the target in the image system; Step 3, after the multi-rotor unmanned aerial vehicle with visual sensor finds the target of interest, the multi-rotor unmanned aerial vehicle with visual sensor hovers, keeps the position and attitude unchanged, calculates the position of the point in front of the unmanned aerial vehicle in the global coordinate system based on the current position and attitude in the global coordinate system, and sends the position and the current attitude angle to a multi-rotor unmanned aerial vehicle without visual sensor in the cluster through the communication middleware, Step 4, after receiving the position and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor flies to the position in the global coordinate system, and adjusts the attitude in the flying process to keep consistent with the multi-rotor unmanned aerial vehicle with visual sensor; The second stage Step 5, the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor can simultaneously obtain the multi-rotor unmanned aerial vehicle without visual sensor and the target of interest, calculates the image deviation of them in the image coordinate system, converts the deviation to the body coordinate system, and then sends the deviation to the multi-rotor unmanned aerial vehicle without visual sensor through the communication network, Step 6, after receiving the image deviation and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the multi-rotor unmanned aerial vehicle without visual sensor calculates the linear velocity in the global coordinate system and the angular velocity in the body coordinate system through the PID controller respectively, and then calculates the rotating speed of each motor by using the position and attitude controller of the multi-rotor unmanned aerial vehicle, so that the attitude angle of the multi-rotor unmanned aerial vehicle without visual sensor keeps consistent with the multi-rotor unmanned aerial vehicle with visual sensor, and moves to the center of the target of interest, thereby realizing the cooperative guidance control of the heterogeneous cluster; Step 7, the cooperative guidance control of other multi-rotor unmanned aerial vehicles without visual sensor in the cluster is realized by repeating the above steps 3 to 6.

2. The vision-based cooperative control method of heterogeneous multi-copter UAV swarm according to claim 1, wherein, Specifically comprising the following steps: I. The first stage Step 1, the UAV cluster is powered on, each UAV in the UAV cluster can obtain the position and attitude in the global coordinate system in real time, the multi-rotor UAV with visual sensor starts working, and the multi-rotor UAV without visual sensor waits for the instruction sent by the multi-rotor UAV with visual sensor; Step 2, the multi-rotor UAV with visual sensor searches for the target of interest that needs to be tracked and crossed, checks whether the target of interest exists by using the real-time image obtained by the visual sensor through the target recognition algorithm of the deep neural network, and calculates the center position of the target in the image system; Step 3: After the multi-rotor UAV equipped with a vision sensor detects a target of interest, it hovers, maintaining its position and attitude, based on its position and attitude in the current global coordinate system. Calculate the coordinates of the body The position of the point in front of this drone in the global coordinate system Will The current attitude angle is sent via communication middleware to a multi-rotor drone in the cluster that is not equipped with a visual sensor. The calculation method is as follows: In formula (1), is the roll angle of the multi-rotor unmanned aerial vehicle with vision sensor installed in the global coordinate system t is the pitch angle of the multi-rotor unmanned aerial vehicle with vision sensor installed in the global coordinate system t is the yaw angle of the multi-rotor unmanned aerial vehicle with vision sensor installed in the global coordinate system t is the rotation matrix of the body coordinate system to the global coordinate system constructed by the roll angle is the translation matrix of the body coordinate system to the global coordinate system, x t , y t , z t are respectively the components of the multi-rotor unmanned aerial vehicle with vision sensor installed in the global coordinate system on the x, y, z three coordinate axes is a point in front of the multi-rotor unmanned aerial vehicle with vision sensor installed, L x , L y , L z are the components of the point on the x, y, z axes in the body coordinate system Step 4, the multi-rotor unmanned aerial vehicle without visual sensor installed receives and the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor installed, flies to the point under the global coordinate system This point, and adjust the attitude during flight, consistent with the multi-rotor unmanned aerial vehicle with visual sensor installed; II. Second stage Step 5, the visual sensor of the multi-rotor unmanned aerial vehicle with visual sensor can acquire the multi-rotor unmanned aerial vehicle without visual sensor and the target of interest at the same time, and calculate the image deviation of them in the image coordinate system Then Convert to the body coordinate system Then send to the multi-rotor unmanned aerial vehicle without visual sensor through the communication network, And The calculation method is as follows: In formula (2), (u t ,v t ) is the center position of the target of interest in the image coordinate system; (u q ,v q ) is the center position of the multi-rotor unmanned aerial vehicle without the visual sensor in the image coordinate system; in formula (3), f is the focal length of the visual sensor, and is the intrinsic parameter of the visual sensor, which is a known constant; Step 6, multi-rotor unmanned aerial vehicle without visual sensor based on received image deviation And the attitude angle of the multi-rotor unmanned aerial vehicle with visual sensor, the linear velocity in the global coordinate system and the angular velocity in the body coordinate system of the multi-rotor unmanned aerial vehicle without visual sensor are calculated through the PID controller respectively, and then the speed of each motor is calculated by using the position and attitude controller of the multi-rotor unmanned aerial vehicle, so that the attitude angle of the multi-rotor unmanned aerial vehicle without visual sensor is consistent with that of the multi-rotor unmanned aerial vehicle with visual sensor, and the center of the target of interest is moved, thereby realizing the cooperative guidance control of the heterogeneous cluster. The PID controller of the linear velocity in the body coordinate system is designed as: In formula (4), and PID controller of the linear velocity, the proportion, integral and differential parameters need to be parameterized; The PID controller of the angular velocity in the body coordinate system is designed as: In formula (5), φ t ′ , θ t ′ , ψ t ′ are the roll angle, pitch angle and yaw angle respectively without the visual sensor installed; and are the proportional, integral and derivative parameters of the PID controller of the angular velocity, which need to be parameter tuned; v b φ in the global coordinate system t ′ θ t ′ ψ t ′ Constructed rotation matrix Transforming to the global coordinate system, the formula for calculating linear velocity in the global coordinate system is: In formula (6), v w is the speed control instruction of the multi-rotor unmanned aerial vehicle in the global coordinate system without installing a visual sensor r b , v w A position and attitude controller is given to the multi-rotor unmanned aerial vehicle, the rotation speed of each motor is calculated by the position and attitude controller, the motion control of the unmanned aerial vehicle is realized, the attitude angle of the multi-rotor unmanned aerial vehicle without the visual sensor is consistent with that of the multi-rotor unmanned aerial vehicle with the visual sensor, and the center of the target of interest is moved, so that the cooperative guidance control of the heterogeneous cluster is realized. Step 7, repeat the above steps 3 to 6 to realize the cooperative guidance control of other multi-rotor UAVs without visual sensor in the cluster.

3. The vision-based cooperative control method of heterogeneous multi-copter UAV swarm according to claim 1, wherein, Each UAV in the cluster can calculate its position and attitude in the global coordinate system in real time through the motion capture system or multi-sensor fusion positioning technology, and send its position and attitude to other UAVs in the cluster through the communication middleware, and obtain the position and attitude of other UAVs in the global coordinate system in real time.

4. The vision-based cooperative control method of heterogeneous multi-copter UAV swarm according to claim 1, wherein, The required coordinate systems are global coordinate system, body coordinate system and image coordinate system, wherein the global coordinate system and the body coordinate system are right-handed systems, the image coordinate system represents the projection of a three-dimensional object on the image plane, the origin is the upper left corner of the image, the u-axis is parallel to the horizontal right of the image plane, and the v-axis is perpendicular to the u-axis downward, the x-axis direction of the global coordinate system and the body coordinate system is consistent with the direction of the visual sensor optical axis, the origin of the global coordinate system is determined by the cluster positioning method, and the origin of the body coordinate system is the optical center position of the visual sensor.

5. The vision-based cooperative control method of heterogeneous multi-copter UAV swarm according to claim 1, wherein, The position and attitude of the UAV cluster in the global coordinate system are provided by the motion capture system, the real-time center position of the balloon and the multi-rotor UAV without visual sensor in the image coordinate system is calculated by using the target recognition algorithm of yolov5 based on deep neural network, the UAVs in the cluster communicate through wireless routing, and the position and attitude controller of the multi-rotor UAV is PX4.

Images