Cooperative control system and method based on heterogeneous unmanned aerial vehicles

Abstract

The invention relates to a cooperative control system and method based on heterogeneous unmanned aerial vehicles. The control system comprises a ground station control system and a mobile terminal control system. The ground station control system is a desktop application supporting task issuing to a plurality of mobile terminals; the mobile terminal is an Android application and can directly control the flight of the single unmanned aerial vehicle through the unmanned aerial vehicle controller; the method comprises the following steps: establishing a relation among the ground station, the mobile terminal and the heterogeneous unmanned aerial vehicle to carry out image return. The heterogeneous unmanned aerial vehicle cooperative control system and method relate to technologies and systems in multiple aspects such as task analysis, communication protocols, data fusion, state monitoring and autonomous cooperation, effective cooperation among heterogeneous unmanned aerial vehicles is achieved through system design and application, and the application potential of the unmanned aerial vehicles in the military and civil fields is expanded.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle technology control, and in particular to a heterogeneous unmanned aerial vehicle collaborative control system and method. Background Art

[0002] With the rapid development of UAV technology in recent years, UAVs are playing an increasingly important role. They are widely used in both military and civilian fields. In terms of actual application scenarios, a single UAV is usually unable to complete the given task, and its capabilities can no longer meet the task requirements. Therefore, the task coordination of multi-UAV systems has become the cutting-edge development direction in the current control field. Through the collaborative work between UAVs of different types or functions, and the control system that completes complex tasks, more flexible and efficient task execution has become a new focus. The ground station plays an important role in the heterogeneous UAV collaborative control system and is a key platform for commanding and controlling UAV flight and task execution. Users on mobile terminals (such as mobile phones and tablets) can monitor the status, flight trajectory and task execution of UAVs in real time, and realize mobile operation and control, which is convenient and fast.

[0003] However, there are also some problems with the existing technology. First, there is no ground station designed for multi-UAV collaborative control. Existing UAV ground station control platforms, such as QGroundControl, Ardupilot, etc., only support the control of a single machine. In practical applications, due to the lack of multi-machine collaborative control functions, the realization of multi-machine collaborative operations requires the use of additional planning software to generate waypoints for each machine, and then multiple ground stations execute or perform corresponding tasks separately or multiple times, so it is impossible to directly complete multi-machine collaborative operations in the same ground station. Second, only specific models of UAVs are supported, heterogeneous UAV multi-machine collaboration is not supported, and the power differences between different UAVs are not considered during planning. At present, some ground stations dedicated to UAV modeling, such as DJI Zhitu, mainly support specific models of UAVs, and have limitations in multi-machine collaborative operations, especially in the planning stage. The power limit of the UAV is not fully considered, resulting in the planned flight path may exceed the actual executable range, thereby affecting the overall execution efficiency and success rate of the task. Third, it does not support the return of aerial images, or the return of images will cause a long-term blockage of the UAV flight. Since drone aerial modeling requires high-quality aerial images, and sending back high-quality aerial images will block the flight of the drone, existing drone modeling applications often require the drone to return and retrieve the aerial images for modeling. This results in the existing aerial modeling requiring detailed image collection of all areas at once, which is not flexible enough and has a poor user experience. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, provide a system based on heterogeneous UAV collaborative control, and provide a method based on heterogeneous UAV collaborative control.

[0005] The main purpose of the present invention is achieved through the following technical solutions: a collaborative control system based on heterogeneous UAVs, including a ground station control system and a mobile terminal control system; the ground station control system is used to issue tasks to multiple mobile terminals, and the mobile terminal control system is used to directly manipulate and control the flight of the corresponding UAVs; through the heterogeneous UAV communication function module of the ground station control system, the instructions and communications between the ground station and the mobile terminal are realized, and by setting a compatible middleware on the mobile terminal, the mobile terminal system can parse the custom instructions sent by the ground station into control instructions for the corresponding UAV, and convert the status information of different UAVs into a unified custom format, thereby realizing the unified control and monitoring of the real-time status of various heterogeneous UAVs by the ground station.

[0006] The control method based on the heterogeneous UAV cooperative control system, the specific steps of the control method are as follows:

[0007] Step 1: Establish communication connections between heterogeneous drones, mobile terminal control systems, and ground station control systems;

[0008] Step 2: After the connection is established, the mobile terminal control system monitors the real-time latitude and longitude, remaining power, and camera video stream of the drone through the status monitoring submodule and displays them on the client interface of the mobile terminal, and sends this information to the ground station at the same time;

[0009] Step 3: The ground station performs path planning, interprets the path planning results into waypoint shooting tasks for each drone, and sends them to the mobile terminal;

[0010] Step 4: After the mobile terminal receives the waypoint shooting task file, it sets the waypoint task according to the latitude and longitude information in the path planning, and then sends it to the drone, and the drone starts to execute the task;

[0011] Step 5: When each drone reaches a waypoint target, the mobile terminal pauses the waypoint access mission, controls the camera on the drone, adjusts it to the angle specified in the mission message, takes a photo, and notifies the image processing function module that a new image has been taken;

[0012] Step 6: The image processing function module receives the message. When the number of images that the drone has not downloaded to the mobile terminal reaches a certain threshold, it starts to try to batch download all the undownloaded images to the mobile terminal; after the download is completed, the execution of the waypoint mission is resumed;

[0013] Step 7: Every time a picture is downloaded to the mobile terminal, it will be uploaded to the ground station independently and in parallel. Every time the ground station receives an aerial image, it will start the aerial image stitching task, complete the image stitching based on the aerial image, longitude and latitude and drone number information, and output the overview of the current area to be modeled.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1. The communication link of ground station-mobile terminal-UAV adopted in the present invention supports the ground station to establish connection with multiple mobile terminals and communicate with each other. Each mobile terminal directly establishes connection with the corresponding UAV and realizes control, status monitoring and data collection for the connected UAV. Such a structure realizes the coordinated control and monitoring of multiple UAVs by the ground station. In addition, compatible middleware is introduced in the mobile terminal. By designing the middleware UAVAgent, TCP message transmission and command conversion between the ground station and heterogeneous UAVs of different flight controllers are realized, ensuring that the ground station can uniformly control and monitor the real-time status of various types of UAVs.

[0016] 2. In the heterogeneous UAV collaborative control method of the present invention, an energy consumption prediction module is introduced into the path planning to predict the flight distance of the heterogeneous UAV, and a round-trip route is planned based on the flight distance constraints and the area to be modeled. Waypoint sampling is performed on the basis of the planned route combined with the heading and lateral overlap rate data set by the image acquisition task, and the waypoints are allocated to different UAVs using a clustering method. The energy consumption prediction function based on deep learning ensures the feasibility of the planning task for each heterogeneous UAV, and the clustering allocation of waypoints ensures the efficiency of task execution.

[0017] 3. In the heterogeneous UAV collaborative control method of the present invention, during the image processing process, it supports two strategies of returning the original aerial image and the aerial thumbnail. It also supports setting the return interval to reduce the impact of the UAV's need to suspend flight when returning pictures on the mission. In addition to returning thumbnails of aerial images, the Beidou high-precision positioning data obtained synchronously with the picture shooting will also be returned. This data can be used to generate a rough overview of the area to be modeled. Based on the overview information, the areas that need to be modeled can be selected for more refined oblique photography image acquisition, thereby improving the efficiency and quality of generating overview maps of the target area, which is more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a composition diagram of a heterogeneous UAV collaborative control system according to the present invention;

[0019] Figure 2 This is a composition diagram of a ground station control system based on a heterogeneous UAV collaborative control system of the present invention;

[0020] Figure 3 This is a composition diagram of a mobile terminal control system based on a heterogeneous UAV collaborative control system of the present invention;

[0021] Figure 4 The present invention is a workflow diagram of a method for collaborative control of heterogeneous UAVs. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] The implementation of the present invention is described in detail below in conjunction with a specific embodiment. This embodiment is based on DJI's heterogeneous drone, and this method is applicable to other types of heterogeneous drones except DJI.

[0024] like Figure 1 As shown, the present invention proposes a collaborative control system based on heterogeneous UAVs, including multiple heterogeneous UAVs that perform tasks, a ground station control system, and a mobile terminal control system. The ground station control system (referred to as the ground station) is a desktop application that supports issuing tasks to multiple mobile terminals, and the mobile terminal control system (referred to as the mobile terminal) is an Android application that is used to directly manipulate and control the flight of the corresponding UAV. The ground station control system can receive the path file of the path planning, distribute the shooting tasks to each UAV for execution, and transmit back the status of the UAV in real time when the UAV is flying.

[0025] like Figure 2 As shown, the ground station control system mainly includes a heterogeneous UAV communication functional module, a heterogeneous UAV path planning functional module, a task analysis functional module, an image processing functional module and a video streaming functional module.

[0026] (1) Heterogeneous UAV communication function module;

[0027] The heterogeneous UAV communication function module is mainly responsible for supporting the instructions and communications between the ground station and the mobile terminal. Most of the existing ground station control systems adopt the solution of directly establishing a communication link with a single UAV, which results in that they can only realize the control and status monitoring of a single UAV, and because the communication instructions of different UAV flight control systems are different, they only support the flight control of a specific model of UAV. In order to realize the multi-machine collaborative control of the ground station, the present invention adopts a communication link of ground station-mobile terminal-UAV. The mobile terminal directly establishes a communication and control link with the corresponding UAV, and the ground station establishes connections with multiple mobile terminals. The communication protocol that is independent of the specific device can be defined on the ground station side, and the instructions are sent to different mobile terminals. The mobile terminal decodes and converts them into the instruction format that the corresponding UAV can support, which solves the compatibility problem between different flight control systems and realizes efficient multi-machine collaborative control.

[0028] The heterogeneous UAV communication function module includes: communication submodule, control function submodule, and network stability control submodule;

[0029] 1) Communication submodule: Design a standardized and unified communication protocol, use the TCP protocol to transmit messages, and establish communication between the ground station and the UAV mobile terminal through socket technology. The ground station uses a system-defined platform-independent communication protocol based on the JSON text format. This module establishes a connection with the mobile terminal through the ground station, sends a customized message format to the mobile terminal, and the mobile terminal parses and converts it into an interface that can be called by the heterogeneous UAV SDK to realize the control of the UAV.

[0030] 2) Control function submodule. This module can transmit information such as configuring drone settings, setting up electronic fences, controlling drone takeoff and landing, and executing waypoint missions. In this way, the ground station can flexibly control drones of different types and configurations.

[0031] 3) Network stability control submodule: In view of the problem that the TCP protocol cannot detect the disconnection in time, the module uses the heartbeat packet mechanism to detect the network disconnection and automatically attempts to reconnect to ensure the continuity and stability of communication.

[0032] (2) Heterogeneous UAV path planning function module:

[0033] The heterogeneous UAV path planning function module is mainly responsible for planning the flight trajectory of heterogeneous UAVs based on the mission scenarios given by users.

[0034] (3) Task analysis function module;

[0035] Receive the path planning results sent by the heterogeneous UAV path planning function module and convert them from the path files of multiple UAVs into flight mission messages for each UAV. In order to deal with some special situations, such as: some UAVs fail to fully complete the assigned waypoint tasks, some UAVs fail to respond to waypoint mission messages, etc., the task parsing module provides the function of splitting the path planning results. You can select the routes of some UAVs and save them as files for independent execution, or you can intercept some path points from the complete route of each UAV, so as to assign tasks to UAVs more flexibly and meticulously.

[0036] (4) Video streaming function module;

[0037] Use RTMP protocol to receive real-time video stream from drones. You can choose to play the video stream of any drone camera, and can capture frames in the video stream and save them as pictures.

[0038] (5) Image processing function module;

[0039] Image processing function module, used to realize batch transmission and stitching of aerial images;

[0040] The existing UAV control system only provides real-time video streaming data transmission for the operator to observe the flight status of the UAV, and the aerial images collected at fixed points need to be manually downloaded from the SD card after the UAV completes the flight mission to support subsequent tasks. The pictures of the present invention are taken by the UAV and stored on the SD card of the UAV. A certain threshold is set. After accumulating to a certain threshold, the picture processing function module actively downloads it to the mobile terminal and uploads it to the ground station.

[0041] Specifically, heterogeneous drones can transmit back two types of images: thumbnails and original images. The present invention uses batch downloading to alleviate the impact of transmitted images on flight congestion. The image processing function module counts the number of images that have not been downloaded, sets a threshold, and after reaching the threshold, several images are downloaded together. After the image is downloaded to the mobile terminal, the mobile terminal starts an independent thread and uses the HTTP protocol to upload the image to the ground station in parallel, without a bottleneck in the rate. The images downloaded to the ground station include the original image data, the data source (the number of the drone taking the photo), and the longitude and latitude of the aerial image.

[0042] The image processing function module can obtain the image information list on both the drone and the mobile device, and distinguish which images are taken for this aerial photography mission and which images are to be downloaded.

[0043] In addition, the refined 3D reconstruction based on the original aerial image will be very time-consuming and cannot quickly guide the next flight plan. The image processing function module will output the stitching result of the currently acquired aerial images based on the aerial image information transmitted back, through feature extraction, feature matching, global optimization and image fusion, and determine the longitude and latitude parameter equation of the stitching image based on the longitude and latitude information of all images to achieve the rough positioning of the longitude and latitude of pixels in the stitching image, so as to realize the interactive key area selection function based on the real-time stitching overview map.

[0044] Since it is impossible to understand the sub-areas that should be focused on in the area to be modeled based on the offline map, the image processing function module receives and stitches aerial images from different drones to obtain an overview of the area to be modeled. Based on the overview, a graphical interactive method is provided to select the areas that need to be photographed on the stitched image, and further collect pictures of the key areas to facilitate the later fine modeling.

[0045] The mobile terminal control system of the heterogeneous UAV collaborative control system mainly includes a UAV connection module, a command parsing module, a waypoint shooting task module, a status monitoring module and a compatible middleware module.

[0046] (1) UAV connection module;

[0047] Establish a communication connection between the drone and the mobile terminal; at the same time, confirm whether the mobile terminal is connected to the drone and whether the mobile terminal application has logged in to the DJI account to unlock the normal flight range. When everything is ready, the application obtains the drone device information, initializes, and allows subsequent operations. The heterogeneous drone communication module of the ground station establishes a connection between the ground station and the mobile terminal, while the drone connection module of the mobile terminal establishes a communication link with the corresponding drone. Through these two communication links, an indirect connection between the ground station and the drone is realized.

[0048] (2) Command parsing module;

[0049] The communication connection between the drone and the mobile terminal is established according to the drone connection module. After parsing the planned multi-machine aerial photography tasks, the task analysis module of the ground station sends different instructions to the corresponding mobile terminals. The command analysis module of the mobile terminal parses the commands sent from the ground station and converts them into control instructions that the corresponding drone can receive, thereby executing tasks such as setting up flight and starting waypoint shooting.

[0050] (3) Waypoint shooting task submodule;

[0051] Set the waypoint according to the latitude and longitude information in the ground station command to start the waypoint mission. When each drone reaches each waypoint, suspend the waypoint mission, adjust the drone camera angle to take photos, and then resume the waypoint mission. Monitor the drone's execution of the waypoint shooting mission, display the mission execution status on the terminal interface, and upload it to the ground station. In order to deal with unexpected situations where the waypoint shooting mission cannot be successfully completed once, the waypoint shooting mission module provides a breakpoint resume function. The waypoint shooting mission will cache the mission information and record the waypoint mission execution progress, and continue the mission execution after the mission terminal.

[0052] (4) Status monitoring module;

[0053] By calling the corresponding interface of DJI MSDK, we can monitor the real-time latitude and longitude, remaining power, camera video stream and other information of the drone, display this information on the client interface of the mobile terminal, and send this information to the ground station according to our customized message format encoding, and stream the video stream to the ground station through the RTMP protocol. In addition, the Beidou navigation module is not an internal module of the drone. It is mounted on the drone's onboard computer. The onboard computer needs to read the Beidou navigation coordinates and obtain Beidou satellite information through the communication between the mobile terminal and the onboard computer using DJI MSDK and OSDK.

[0054] The existing UAV control system only provides real-time video streaming data transmission for operators to observe the flight status of the UAV. Aerial images collected at fixed points need to be manually downloaded from the SD card after the UAV completes the flight mission to support subsequent tasks.

[0055] 5) Compatible middleware module. UAVAgent is introduced as a middleware to receive control commands from the ground station and monitor the real-time status of heterogeneous drones. UAVAgent can parse the standard command format sent by the ground station and convert it into commands that can be received by the current heterogeneous drone flight controller SDK. At the same time, the heterogeneous drone status information provided by different flight control SDKs is converted into standard data types that can be received by the ground station, realizing the standardized and unified processing of heterogeneous drone data.

[0056] After completing the design of the above heterogeneous UAV cooperative control system, the present invention also discloses a method for cooperative control of heterogeneous UAVs, such as Figure 3 As shown, the specific steps are as follows:

[0057] Step 1: Establish communication connections between heterogeneous drones, mobile terminal control systems, and ground station control systems;

[0058] Step 1.1: The mobile terminal detects whether there is a legitimate drone connected to the device, and if so, allows the subsequent process to proceed.

[0059] Step 1.2: The mobile terminal actively initiates a request to establish a network connection with the ground station control system.

[0060] Step 2: After the connection is established, the mobile terminal control system monitors the real-time latitude and longitude, remaining power, and camera video stream of the drone through the status monitoring submodule and displays them on the client interface of the mobile terminal, and sends this information to the ground station at the same time.

[0061] Step 3: The ground station performs path planning, parses the path planning results into waypoint shooting tasks for each drone, and sends them to the mobile terminal.

[0062] In the existing UAV path planning technology, the path design is usually performed for a single type of UAV cluster. However, due to differences in battery capacity, flight control efficiency, etc., heterogeneous UAV clusters have significant differences in their flight distances. The path planning method of the heterogeneous UAV path planning function module is:

[0063] Step 3.1: Sampling waypoints: The module first generates the waypoints of the drone through a uniform sampling method according to the heading and lateral overlap rate set by the user.

[0064] Step 3.2: Predict flight distance. The energy consumption prediction module based on deep learning is used to predict the flight distance that heterogeneous drones can support under different weather conditions. The type of drone, battery power, and weather conditions are used as input parameters, and the pre-trained multi-layer perceptron model is used to predict the flight distance of each drone.

[0065] Step 3.3: Perform waypoint clustering. According to the number of drones N, all waypoints are clustered into N different clusters using the K-Means algorithm based on Manhattan distance.

[0066] Step 3.4: Perform path allocation. The allocation method is: assign the cluster with the least number of waypoints to the UAV with the shortest flight distance. The path planning problem is transformed into a traveling salesman problem, and the flight path of the UAV is generated by the simulated annealing algorithm.

[0067] Step 3.5: Perform path optimization. Check the generated flight path. If the path length exceeds the predicted flight distance of the drone, delete waypoints from the end of the path until the total path length meets the flight distance limit. At the same time, redistribute the deleted waypoints to the nearest unassigned cluster.

[0068] Step 3.6: Iterative path planning. Continue to plan paths for the cluster with the least number of unassigned waypoints and the UAV with the shortest flight distance until there are no more UAVs to be assigned.

[0069] Step 3.7: Path feasibility check. If all drones have been assigned tasks and there are no uncovered waypoints, the planned path will be sent to the task analysis function module. Otherwise, the system will prompt to increase the number of drones or adjust the heading and lateral overlap rate to reduce the number of waypoints and re-plan the path.

[0070] Step 4: After the mobile terminal receives the waypoint shooting task file, it sets the waypoint task according to the latitude and longitude information in the path planning, and then sends it to the drone, and the drone starts to execute the task.

[0071] Step 5: When each drone reaches a waypoint target, the mobile terminal pauses the waypoint access mission, controls the camera on the drone, adjusts it to the angle specified in the mission message, takes a photo, and notifies the image processing function module that a new image has been taken.

[0072] Step 6: The image processing module receives the message. When the number of images that the drone has not downloaded to the mobile terminal reaches a certain threshold, it starts to try to download all the undownloaded images to the mobile terminal. The image processing module chooses to download thumbnails or original images according to the current flight mission, and resumes the execution of the waypoint mission after the download is completed.

[0073] Step 7: Every time a picture is downloaded to the mobile terminal, it will be uploaded to the ground station independently and in parallel, without blocking other operations. Every time the ground station receives an aerial image, it will start the aerial image stitching task, complete the image stitching based on the aerial image, longitude and latitude, drone number and other information, and output the overview of the current area to be modeled.

[0074] The specific steps for stitching aerial images are as follows:

[0075] Step 7.1: Aerial image data reception; In order to improve the transmission speed of drone aerial images, existing drones usually choose to transmit thumbnails of images, but this will cause the longitude and latitude information in the original image to be lost. However, this information is crucial for subsequent image stitching and analysis. In order to solve this problem, the present invention uses the Beidou navigation module carried by the drone to synchronously obtain high-precision positioning data while transmitting thumbnails and binds the thumbnails to transmit them back to the ground station, thereby retaining the longitude and latitude information of the aerial images.

[0076] Step 7.2: Aerial image feature extraction; The purpose of feature extraction is to find some obvious feature areas in aerial pictures, so as to achieve the association between different aerial pictures. Aerial pictures almost do not have large scale and rotation transformations, but the aerial photography area often has a large span, resulting in obvious changes in lighting. Therefore, the feature point extraction algorithm adopted needs to be able to cope with lighting brightness changes well. The present invention adopts the SURF feature extraction algorithm. The SURF algorithm is an optimization of SIFT and is a robust local feature point detection and description algorithm. It has both the high robustness characteristics of SIFT features and a significant improvement in speed.

[0077] Step 7.3: Aerial image feature matching. There must be a matching relationship between two adjacent aerial images, but due to possible drone control errors (the distance between waypoints is too far), there is no matching relationship between the two continuously received aerial images. The probability of this happening is greater when the heading overlap rate is set to a larger value. In order to enable the final stitching result to cover a sufficiently large area, the aerial image feature matching problem is modeled as a maximum connected graph problem with aerial images as nodes and matching relationships as edges. In addition, the aerial scene is obviously not a linear area. In addition to the overlap between the heading adjacent areas, there will also be overlap between the lateral adjacent areas. In the absence of auxiliary prior information, in order to find the lateral adjacent areas, each newly received picture has to be matched with all the received pictures, which will be very time-consuming. The present invention uses the Beidou high-precision latitude and longitude information corresponding to the aerial images to quickly filter out images that are geographically far away and do not have a corresponding matching relationship, further improving the efficiency of aerial image feature matching.

[0078] Step 7.4: Global optimization. Image stitching is essentially projecting each aerial image onto the final stitching reference image through a transformation matrix. Only by solving the transformation matrix for each image can the final stitching be completed. The transformation matrix of each image is used as the optimization variable, and the error sum between all matching pairs is minimized as the goal (minimizing the distance error sum of all matching feature point pairs when they are transformed to the final reference stitching image through the transformation matrix) is optimized and solved to obtain the transformation matrix of each aerial image.

[0079] Step 7.5: Image fusion. Based on the transformation matrix obtained by minimizing the global matching error module, all images can be rotated to the plane corresponding to the spliced ​​image and fused. Ordinary linear fusion will cause a lot of ghosting in the final splicing result. The present invention adopts the Laplace image fusion algorithm.

[0080] Step 7.6: Interactive area framing. Users can flexibly frame the task area of ​​interest on the fused image. The scan line algorithm automatically converts the framed area into a more regular polygon, and calculates the longitude and latitude coordinates corresponding to the polygon vertex coordinates based on the high-precision Beidou positioning information contained in the image. Finally, the heterogeneous drone path planning function module plans a precise aerial photography path covering the area, and collects more refined images of the area.

[0081] The drones mentioned in the present invention are all heterogeneous drones. The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A collaborative control system based on heterogeneous UAVs, characterized in that: It includes a ground station control system and a mobile terminal control system; the ground station control system is used to issue tasks to multiple mobile terminals, and the mobile terminal control system is used to directly control the flight of the corresponding UAV; Through the heterogeneous UAV communication function module of the ground station control system, the commands and communications between the ground station and the mobile terminal are realized. By setting compatible middleware on the mobile terminal, the mobile terminal system can parse the custom commands sent by the ground station into control commands for the corresponding UAVs, and convert the status information of different UAVs into a unified custom format, thereby realizing the unified control and monitoring of the real-time status of various heterogeneous UAVs by the ground station.

2. The heterogeneous UAV collaborative control system according to claim 1 is characterized in that: The ground station control system includes: Heterogeneous UAV communication function module, used to realize the command and communication between the mobile terminal control system and the ground station; Heterogeneous UAV path planning function module plans the flight trajectory of heterogeneous UAVs based on the mission scenario given by the user; The task parsing function module receives the path planning results sent by the heterogeneous UAV path planning function module and converts the path files of multiple UAVs into flight mission messages for each UAV; Video streaming function module, used to receive real-time video stream from the drone; The image processing function module is used to realize batch transmission and stitching of aerial images.

3. The heterogeneous UAV collaborative control system according to claim 1 is characterized in that: The mobile terminal control system comprises: A drone connection module is used to establish a communication connection between the drone and the mobile terminal; The command parsing module establishes a communication connection between the drone and the mobile terminal according to the drone connection module, and parses the commands sent from the ground station to set the drone address, set the flight mission, and start the waypoint shooting mission; The waypoint shooting task module is used to control each drone to pause the waypoint access task when it reaches each waypoint, adjust the drone camera angle to take photos, and then resume the waypoint task execution; The status monitoring module monitors the real-time latitude and longitude, remaining battery power, and camera video stream of the drone and displays them on the client interface of the mobile terminal, and sends this information to the ground station at the same time; Compatible with the middleware sub-module, it is responsible for uniformly parsing the control instructions of the ground station and monitoring the real-time status of heterogeneous UAVs.

4. The heterogeneous UAV collaborative control system according to claim 2 is characterized in that: The heterogeneous UAV communication function module includes: communication submodule, control function submodule, and network stability control submodule; Communication submodule: Design a standardized and unified communication protocol, use the TCP protocol to transmit messages, and establish communication between the ground station and the UAV mobile terminal through socket technology; the ground station uses a system-defined platform-independent communication protocol based on the JSON text format; the protocol establishes a connection between the ground station and the mobile terminal, sends a customized message format to the mobile terminal, and the mobile terminal parses and converts it into an interface that can be called by the heterogeneous UAV SDK to achieve control of the UAV; The control function submodule can transmit information about configuring drone settings, setting up electronic fences, controlling drone takeoff and landing, and executing waypoint missions; The network stability control submodule uses the heartbeat packet mechanism to detect network disconnection and automatically attempts to reconnect to ensure the continuity and stability of communication in order to solve the problem that the TCP protocol cannot detect disconnection in time.

5. The heterogeneous UAV collaborative control system according to claim 2 is characterized in that: In the image processing function module, during the flight of the drone, by setting a threshold, counting the number of images that have not been downloaded, and downloading the images in batches, the images are downloaded to the mobile terminal to solve the problem of obstruction to the flight caused by the drone transmitting back images.

6. The control method based on heterogeneous UAV cooperative control system according to claim 1 is characterized in that: The specific steps of the control method are as follows: Step 1: Establish communication connections between heterogeneous drones, mobile terminal control systems, and ground station control systems; Step 2: After the connection is established, the mobile terminal control system monitors the real-time latitude and longitude, remaining power, and camera video stream of the drone through the status monitoring submodule and displays them on the client interface of the mobile terminal, and sends this information to the ground station at the same time; Step 3: The ground station performs path planning, interprets the path planning results into waypoint shooting tasks for each drone, and sends them to the mobile terminal; Step 4: After the mobile terminal receives the waypoint shooting task file, it sets the waypoint task according to the latitude and longitude information in the path planning, and then sends it to the drone, and the drone starts to execute the task; Step 5: When each drone reaches a waypoint target, the mobile terminal pauses the waypoint access mission, controls the camera on the drone, adjusts it to the angle specified in the mission message, takes a photo, and notifies the image processing function module that a new image has been taken; Step 6: The image processing function module receives the message. When the number of images that the drone has not downloaded to the mobile terminal reaches a certain threshold, it starts to try to batch download all the undownloaded images to the mobile terminal; after the download is completed, the execution of the waypoint mission is resumed; Step 7: Every time a picture is downloaded to the mobile terminal, it will be uploaded to the ground station independently and in parallel. Every time the ground station receives an aerial image, it will start the aerial image stitching task, complete the image stitching based on the aerial image, longitude and latitude and drone number information, and output the overview of the current area to be modeled.

7. The control method based on heterogeneous UAV cooperative control system according to claim 6 is characterized in that: In step 3, the path planning method of the heterogeneous UAV path planning function module is: Step 3.1: Perform waypoint sampling, and generate the waypoints of the drone through a uniform sampling method according to the heading and lateral overlap rate set by the user; Step 3.2: Predicting flight distance, through the energy consumption prediction module, predict the flight distance of heterogeneous UAVs under different weather conditions, taking the UAV type, battery power and weather conditions as input parameters, and using the pre-trained multi-layer perceptron model to predict the flight distance of each UAV; Step 3.3: Perform waypoint clustering. According to the number of drones N, all waypoints are clustered into N different clusters using the K-Means algorithm based on Manhattan distance. Step 3.4: Perform path allocation. The allocation method is: assign the cluster with the least number of waypoints to the UAV with the shortest flight distance, transform the path planning problem into a traveling salesman problem, and generate the flight path of the UAV through the simulated annealing algorithm; Step 3.5: Perform path optimization and check the generated flight path. If the path length exceeds the predicted flight distance of the drone, delete waypoints from the end of the path until the total path length meets the flight distance limit. At the same time, redistribute the deleted waypoints to the nearest unassigned cluster. Step 3.6: Iterate the path planning and continue to plan the path for the cluster with the least number of unassigned waypoints and the UAV with the shortest flight distance until there are no unassigned UAVs; Step 3.7: Path feasibility check. If all drones have been assigned tasks and there are no uncovered waypoints, the planned path will be sent to the task analysis function module; otherwise, it will prompt to increase the number of drones or adjust the heading and lateral overlap rate to reduce the number of waypoints and re-plan the path.

8. The control method based on heterogeneous UAV cooperative control system according to claim 6 is characterized in that: The steps for stitching aerial images in step 7 are: Step 7.1: Aerial image data reception: while transmitting the thumbnail, the Beidou navigation module on the drone is used to synchronously obtain the positioning data and bind the thumbnail to transmit it back to the ground station, thereby retaining the latitude and longitude information of the aerial image; Step 7.2: Aerial image feature extraction, using the SURF feature extraction algorithm to extract features from the aerial image; Step 7.3: Aerial image feature matching, using the Beidou high-precision latitude and longitude information corresponding to the aerial images to filter out images that do not have a corresponding matching relationship; Step 7.4: Global optimization, taking the transformation matrix of each image as the optimization variable and the minimum value of the sum of errors between all matching pairs as the goal, perform optimization and solve to obtain the transformation matrix of each aerial image; Step 7.5: Image fusion, based on the transformation matrix obtained by minimizing the global matching error module, all images are rotated to the plane corresponding to the stitched image and they are fused using the Laplacian image fusion algorithm; Step 7.6: Interactive area framing. The user can flexibly frame the task area of ​​interest on the fused image. The scan line algorithm automatically converts the framed area into a regular polygon, and calculates the longitude and latitude coordinates corresponding to the polygon vertex coordinates based on the high-precision Beidou positioning information contained in the image. The heterogeneous UAV path planning function module plans a precise aerial photography path covering the area, and collects more refined images of the area.

Images