Unmanned aerial vehicle and unmanned vehicle cross-domain cooperative tracking control method based on YOLOv7

Abstract

The invention discloses an unmanned aerial vehicle and unmanned vehicle cross-domain cooperative tracking control method based on YOLOv7. The method comprises the steps that a user controls an unmanned vehicle to carry the unmanned aerial vehicle to a designated place; the method comprises the following steps: extracting a frame image from a video stream collected by an unmanned aerial vehicle in real time, and processing the frame image to obtain an input image; inputting the input image into a pre-trained YOLOv7 network model to obtain all target information in the input image; a user can specify target information and perform obstacle determination on a non-user-specified target. According to target information in an input image, the system updates a path in real time to avoid obstacles and continuously track a specified target. And the user issues a tracking ending and landing instruction, and the unmanned aerial vehicle searches the target information of the unmanned vehicle in the input image and tracks and lands on the unmanned vehicle. The method can stably operate in a complex and dynamic environment, ensures reliable target tracking, effectively realizes complementation between the unmanned aerial vehicle and the unmanned aerial vehicle, makes full use of respective advantages, and improves the overall system performance.

Description

Technical Field

[0001] The present invention relates to a cross-domain collaborative tracking control method of an unmanned aerial vehicle (UAV) and an unmanned vehicle based on YOLOv7, and belongs to the technical field of air-ground collaboration of UAVs. Background Art

[0002] Drones have the advantages of flexible operation, wide field of view, and strong environmental adaptability, and can effectively perform tasks in complex and dangerous environments. Unmanned vehicles have the advantages of large carrying capacity and high adaptability to complex road environments. Through collaborative perception and control technology, drones and unmanned vehicles dynamically adjust movement strategies according to mission requirements, environmental information and their own status to effectively realize air-ground collaborative tasks. Air-ground collaborative technology effectively makes up for the shortcomings of drones and unmanned vehicles. For example, unmanned vehicles have limited field of view and are usually used for close-range and precise observation of ground targets; and although most drones have a faster movement speed, they are difficult to accurately observe ground targets due to the limitations of load capacity and hovering ability. Therefore, air-ground collaboration has broad application prospects and development potential in military reconnaissance and collaborative operations, disaster relief, logistics and transportation, intelligent transportation, agricultural monitoring and other fields. Summary of the invention

[0003] In view of the above problems of the prior art, the present invention utilizes the complementary advantages between drones and unmanned vehicles and proposes a cross-domain collaborative tracking control method based on YOLOv7. This method achieves long-range, high-endurance and high-precision tracking and monitoring, while having good robustness. Figure 1 The figure is a schematic diagram of the steps of a UAV cross-domain collaborative tracking control method based on YOLOv7 provided by the present invention. The specific method includes:

[0004] S1: The user controls the unmanned vehicle to carry the drone to the designated location;

[0005] S2: extract the frame images from the video stream collected by the drone in real time, and process the frame images to obtain the input image;

[0006] The image acquisition and frame image processing include: the drone acquires video streams, discards some frame images appropriately to increase processing speed, modifies the RGB color channels of the retained frame images to BGR to adapt to YOLOv7 image processing, and then reduces the aspect ratio of the frame images after the color channels are modified to 0.5 times the original to speed up processing. The details are as follows:

[0007] ⑴ Create a local area network: The drone creates a dedicated local area network and adds the terminal device to the local area network to ensure that the terminal device is successfully connected to the drone and use the data transmission protocol to obtain real-time video stream;

[0008] ⑵ Frame image preprocessing: Not all frame images in the acquired video stream are preprocessed. In order to improve the processing speed, when the terminal's processing speed is slow and a large amount of frame image buffer is accumulated or even overflows, the frame image that has newly entered the buffer will be discarded. The specific rules are: when the terminal's frame image buffer occupancy rate exceeds 50%, one frame is discarded for every four frames, when it exceeds 70%, one frame is discarded for every three frames, and when it exceeds 80%, one frame is discarded for every two frames. In this way, the processing speed is improved and the real-time performance is enhanced;

[0009] ⑶ Color channel modification: To adjust the color channel of the retained frame image, you only need to change the bit order of the color vector part. This can be achieved by using the cvtColor function in the OpenCV library and adjusting the encoding format to COLOR_RGB2BGR.

[0010] (4) Image scaling: Reduce the frame image proportionally to 0.5 times of the original size. This can also be achieved using the fast matrix multiplication in the opencv library, which is specifically achieved by using the resize function to set the size to 0.5 times of the frame image.

[0011] S3: Input the input image into the pre-trained YOLOv7 network model to obtain all the target information in the input image, including the target type, confidence and the boundary of the target in the input image;

[0012] The pre-trained YOLOv7 network model (https: / / github.com / WongKinYiu / yolov7) is trained by the following steps: creating a dataset containing category labels, with a total of 1305 images, Figure 2 Some examples of the dataset are shown. The dataset contains labels of unmanned vehicles equipped with drones, tracked objects, and obstacles; and is trained through the following steps:

[0013] ⑴ The training dataset is determined based on the specific usage scenario and tracking target. The characteristics of different scenarios and targets are significantly different, such as in the wild scene and the urban scene, the tracking object is a person or a dog, etc. Even the environment in the wild scene is very different, such as in grassland areas and in forest areas. This embodiment selects indoor scenes, and the tracking object is a person or an unmanned vehicle. Therefore, 10 types of target objects including stools, tables, chandeliers, people, unmanned vehicles, etc. are collected, totaling 1305 labeled training data. The training data contains the target image and its corresponding bounding box and category label.

[0014] ⑵ The training data should cover data under special circumstances to improve the accuracy of model training. Generally speaking, the more extensive the data included in the training data, the higher the accuracy of the model matching the target; if the training set only contains general data, the target is clearly visible in a simple environment, and the model only has a high recognition accuracy in a general environment; however, for edge data, such as targets belonging to the same category but with features that are too different from other similar targets, noisy data, such as blurred frames or blurred targets caused by image acquisition equipment failure, signal loss, and harsh natural environment, and data missing such as packet loss during transmission, the target part is not within the image acquisition range, etc., the model will find it difficult to detect the target information and it is very easy to lose the target information; therefore, the training set of the present invention includes data under various abnormal conditions, and the number of training times is set to 2000 to ensure the effective convergence of the model.

[0015] S4: The user can specify target information and perform obstacle determination for targets not specified by the user. Based on the target information in the input image, the system updates the path in real time to avoid obstacles and continue to track the specified target.

[0016] ⑴DeepSORT, a target tracking and recognition algorithm based on deep learning:

[0017] In S4, the method for the user to specify target information and lock the target adopts the target tracking and recognition algorithm DeepSORT based on deep learning to achieve continuous tracking of multiple targets; DeepSORT is an extension of the SORT (Simple Online and Realtime Tracking) algorithm, which combines deep learning technology to improve the accuracy and stability of multi-target tracking. The algorithm was originally designed for the deep learning version of multi-target tracking. By combining convolutional neural networks (CNN) and Kalman filters, DeepSORT can accurately track multiple objects in a video stream and continue tracking when the objects are occluded or lost. The core of the DeepSORT algorithm is the Kalman filter and the Hungarian matching algorithm. The implementation process of the Kalman filter and the Hungarian matching algorithm in the present invention is described in detail below:

[0018] Kalman filter algorithm In the present invention, the state of each target is represented by an eight-dimensional vector:

[0019]

[0020] Among them, u is the pixel horizontal coordinate of the geometric center point of the target in the image, v is the pixel vertical coordinate of the geometric center point of the target in the image, r is the width of the target boundary, and h is the height of the target boundary. is the pixel lateral velocity of the target, is the pixel longitudinal velocity of the target, is the speed of change of the target boundary width, is the speed of change of the target boundary height, and the speed value of each target state is initialized to 0.

[0021] Kalman filtering is divided into two stages: (1) predicting the position of the target at the next moment, and (2) updating the target position based on the observation results. In the present invention, it is assumed that the target motion state satisfies the uniform motion model:

[0022] (1)

[0023] In formula (1) Indicates the motion state of the target at the previous moment. is the state transfer matrix, noise Assuming it is Gaussian white noise, its noise variance matrix is is a diagonal matrix, and the expression is as follows:

[0024]

[0025] Then the error covariance prediction is calculated as:

[0026]

[0027] At the same time, let the initial value of the state estimation error covariance be a diagonal matrix, and the initial value of the state estimation be the observed value, which have the following expressions:

[0028]

[0029] Build the observation model:

[0030]

[0031] Among them, H k is the observation transfer matrix, v k is the observation noise, which represents the uncertainty of the observation. Assume that v k is Gaussian white noise, and its noise variance matrix is It maps the target state to the detection space, which has the following expression:

[0032]

[0033] Kalman filter gain update:

[0034]

[0035] Status update correction:

[0036]

[0037] Error covariance update correction:

[0038] P′ k =P k -KH k P k (6)

[0039] At this point, the iterative process of the Kalman filter is completed, and the position of the detection target in the image at the current moment is estimated.

[0040] The traditional approach to solving the association between the predicted Kalman state and the newly arrived measurements is to formulate an assignment problem that can be solved using the Hungarian algorithm. In this formulation of the problem, we integrate motion information and appearance information by combining two appropriate metrics. To incorporate motion information, the (squared) Mahalanobis distance between the predicted Kalman state and the newly arrived measurements is used:

[0041]

[0042] In formula (7), d (1) (i, j) is the Mahalanobis distance, d j is the target state currently measured and represented by an eight-dimensional vector, y i is the mean of the target state estimates, S i is the covariance matrix of the Kalman estimated state. The Mahalanobis distance takes the uncertainty of the state estimate into account by measuring how many standard deviations the detected position is from the mean trajectory. In addition, using this measure, we can calculate the uncertainty of the state estimate by using the 95% confidence interval from the inverse distribution The Mahalanobis distance calculated in is used to exclude unlikely associations, and an indicator is used to indicate this decision:

[0043]

[0044] The Mahalanobis distance is a suitable association metric when the motion uncertainty is low, however, in image space, the predicted state distribution obtained in the Kalman filter framework only provides a rough estimate of the object position. In particular, unexplained camera motion introduces fast displacements in the image plane, making the Mahalanobis distance rather inaccurate when tracking occluded objects. Therefore, a second metric is integrated into the assignment problem. For each bounding box detection d j , we obtain its global feature map through the CNN network and then use l 2 Normalize and calculate its appearance descriptor r j And ||r j ||=1. At the same time, for the predicted state of each target, retain I k = 100 associated appearance descriptors, which represent the appearance features of the target in the first 100 frames:

[0045]

[0046] The minimum cosine distance between the i-th predicted state and the j-th measured target state in the appearance space is used as the second metric, which is defined as follows:

[0047]

[0048] Similarly, set a binary indicator for this metric:

[0049]

[0050] Cosine distance is less than or equal to threshold t (2) = 0.6, otherwise it is assigned 1. The two metrics can complement each other by combining different aspects of the assignment problem. On the one hand, Mahalanobis distance provides information about possible object locations based on motion, which helps in short-term prediction. On the other hand, cosine distance takes into account appearance information, which can play a role in restoring identity after long-term occlusion. To formulate the association problem, the two metrics are combined using a weighted sum:

[0051] c i，j =λd (1) (i, j)+(1-λ)d (2) (i, j) (12)

[0052] When there is a large amount of displacement of the drone, the contribution of the target's motion information to the target tracking will be greatly reduced, but the target's appearance characteristics are not affected by the drone's motion. Taking all factors into consideration, the present invention gives λ = 0.1. If the two indicators are within the gated area, we call it an acceptable correlation:

[0053]

[0054] (2) Mathematical model of UAV tracking control method:

[0055] We use the above algorithm to obtain the accurate position information of the target in the input image. We use this information to adjust the flight attitude of the drone. The flight attitude includes: v(s) the forward speed of the drone and θ(d) the yaw angular velocity:

[0056]

[0057] Among them, v(s) is the forward speed of the UAV, s is the distance between the target and the UAV, which is obtained by using the optical flow method through visual information, and v 0 is the set starting speed, s max is the maximum distance threshold, s minis the minimum distance threshold, and a is the increasing coefficient; it can be seen from formula (14) that as the target moves away (or approaches) quickly, the speed of the drone will change linearly quickly to maintain an appropriate distance from the target. In order to facilitate fast calculation and better robustness, independent judgment criteria and conditions are set for the speed vector and angular velocity vector of the drone. Obviously, it is difficult to achieve the goal of continuous locking and tracking by only changing the speed value of the drone, so it is still necessary to introduce the drone yaw angular velocity model. The velocity model of the drone yaw angle is as follows:

[0058]

[0059] e d =dd mid

[0060] in, is the yaw angular velocity, d, d safe ,d mid are the position of the target center in the input image, the safety threshold of the target center's deviation from the input image center, and the position of the input image center; a and b are increasing coefficients, e is d ′ is the deviation value of the previous moment. In formula (15), in order to make the change of the UAV heading smoother and avoid excessive turning amplitude when approaching the target center position, the deviation value measured at the previous moment is introduced into the current moment to produce a negative feedback effect. At the same time, the intensity of this negative feedback is limited by setting the coefficient.

[0061] (3) Speed ​​model of drone obstacle avoidance control method:

[0062] What micro UAVs need more in complex scenes such as cities or forests is the ability to respond to various emergencies. In order to meet this demand, independent judgment criteria and conditions are set for the speed vector and angular velocity vector of the UAV. Specifically, for non-user-specified targets, computer vision is used to estimate the distance from the UAV in three-dimensional space according to different target types and boundaries; a safety distance is set, and non-specified targets whose distance from the UAV is greater than the safety distance are not considered; for each non-specified target whose distance is less than the safety distance, we establish a three-dimensional space speed obstacle cone, and combine the size of the UAV itself to calculate whether the UAV will collide with the non-specified target during tracking; in the face of possible collisions, respond to fast obstacles and establish the following speed model:

[0063]

[0064] In formula (16), u(x) is the lateral velocity of the UAV, x, x safeThey are the lateral distance (the right direction is defined as the positive direction) and the lateral safety distance between the drone and the nearest obstacle in three-dimensional space. Similarly, the above model can also be applied to the vertical climbing speed of the drone (the vertical ground upward is defined as the positive direction):

[0065]

[0066] In formula (17), n(y) is the vertical speed of the UAV, y, y safe They are the vertical distance and vertical safety distance between the drone and the nearest obstacle in three-dimensional space. The overall control of the drone is completed by combining these vectors. The algorithm flow is as follows: Figure 4 As shown, the three-dimensional space speed barrier cone is explained below:

[0067] Velocity Obstacle (VO) is a conical obstacle interval generated in the velocity space of the drone. As long as the relative velocity vector is outside the conical area, the drone will not collide with the obstacle in the future. In three-dimensional space, assuming that the radiation radius of the drone is R u , the radiation radius of a dynamic threat obstacle is R o At a certain moment, the positions of the drone and the obstacle are P u , P o In the calculation, the position of the UAV is usually taken as the origin, and the position of the obstacle can be estimated from the visual information of the UAV; the velocity vector of the UAV is V u ; The obstacle velocity vector is V o ; UAV’s position information (P u , V u ) and the position information of obstacles (P o , V o ) can be expressed by formula (18) to formula (21):

[0068] P u =(x u ,y u , z u ) (18)

[0069]

[0070] P o =(x o ,y o , z o ) (20)

[0071]

[0072] where v u 、v oare the velocity vector magnitudes of the drone and the obstacle respectively; θ u , and θ o , are the velocity vector direction angles of the UAV and the obstacle (θ is the pitch angle, is the yaw angle); Figure 3 As shown, P o As the center of the sphere, R is the radius to make a sphere with P u Make a cone and a sphere P as the vertex o Tangent, the sphere can better simulate dynamic obstacles with strong maneuverability such as airplanes; uo is the relative vector velocity of the two, and the magnitude of the relative velocity is v uo , d is the Euclidean distance between the drone and the obstacle, and their relationship is as follows:

[0073] R=R u +R o (26)

[0074] V uo =V u -V o (27)

[0075]

[0076] If the vector V uo If the end is within the three-dimensional speed obstacle cone, the dynamic obstacle poses a threat to the drone. For these obstacles that pose a threat to the drone, the priority is divided according to the distance from the drone. The closer the distance, the higher the priority. The drone will avoid the obstacle with the highest priority and then repeat the avoidance process.

[0077] S5: The user issues a command to end tracking and land. The UAV searches for the target information of the unmanned vehicle in the input image, tracks and lands on the unmanned vehicle.

[0078] Tracking landing control method:

[0079] First, the unmanned vehicle is identified and detected by the YOLOv7 network model to obtain the target information of the unmanned vehicle, and the position of the unmanned vehicle relative to the drone is estimated based on the pixel ratio of the unmanned vehicle target boundary in the input image and the preset unmanned vehicle distance ratio coefficient; secondly, the drone controls the flight attitude to fly near the unmanned vehicle and turns on the downward camera of the drone. The video stream obtained by the downward camera is processed and input into the YOLOv7 network model, and YOLOv7 recognizes the landing sign above the unmanned vehicle; finally, the parallel PID control algorithm is used to control the speed of the drone, that is, to keep the relative position of the drone and the unmanned vehicle on the horizontal plane unchanged, and at the same time, the drone controls the altitude to gradually decrease until it lands on the unmanned vehicle.

[0080] At this point, the drone can realize cross-domain collaborative tracking functions between drones and unmanned vehicles in dynamic and complex environments based on YOLOv7 and control algorithms.

[0081] Compared with the prior art, this paper proposes a cross-domain collaborative tracking control method for UAVs and unmanned vehicles based on YOLOv7, aiming to achieve detection and tracking of non-cooperative targets in general environments. This method relies on the obstacle avoidance algorithm to significantly improve the environmental adaptability of UAVs. Its main advantages include:

[0082] 1) High robustness: Able to operate stably in complex and dynamic environments, ensuring reliable target tracking.

[0083] 2) Air-ground collaboration: effectively realize the complementarity between drones and unmanned vehicles, make full use of their respective advantages, and improve the overall system performance.

[0084] 3) Diversified applications: It is suitable for a variety of mission scenarios, such as military reconnaissance, logistics and transportation, disaster relief, etc., showing broad application potential.

[0085] Through these advantages, the present invention provides an innovative solution for the collaborative operation of drones and unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a flow chart of the UAV cross-domain collaborative tracking control method based on YOLOv7.

[0087] Figure 2 This patent provides a schematic diagram of establishing a three-dimensional space speed barrier cone.

[0088] Figure 3 It is a labeled dataset that is partially manually acquired and manually annotated.

[0089] Figure 4 It is the execution process of the drone tracking target and avoiding obstacles control.

[0090] Figure 5 It is a diagram of the implementation effect of the method from the perspective of a drone and a diagram of a drone tracking an unmanned vehicle from a third-person perspective.

[0091] Figure 6 It is the F1 score generated when training the YOLOv7 network model to evaluate the training effect

[0092] The symbols in the figure are explained as follows:

[0093] Figure 3 Explanation of symbols in: R u is the radiation radius of the drone, R o is the radiation radius of the dynamic threat obstacle, Pu , P o are the positions of the drone and the obstacle, V u is the UAV velocity vector, V o is the obstacle velocity vector, V uo is the relative vector velocity between the two, R is the radius of the obstacle sphere, d is the Euclidean distance between the drone and the obstacle, α 0 This is the drone's spherical view of the obstacle. DETAILED DESCRIPTION

[0094] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0095] This embodiment uses the DJI Tello micro multi-rotor drone and the Lingao Technology MiniUGV20C four-wheel drive suspension car as examples. The relevant technical parameters are shown in the following table:

[0096] Table 1 Key threshold settings

[0097] <![CDATA[s min ]]> 0.5m <![CDATA[s max ]]> 2m <![CDATA[d save ]]> 50pixels <![CDATA[d mid ]]> 240 pixels <![CDATA[x save ]]> 0.4m <![CDATA[y save ]]> 0.4m

[0098] (1) After gaining an in-depth understanding of the types of objects and organisms that may appear in the application scenario, we obtained training data sets with labeled data of these types. The source of this data set can be a public database, data manually labeled after field collection, etc. The experimental scene of this example is set indoors. The tracking target is specified as an unmanned vehicle carrying a drone. The obstacles are common indoor objects such as tables and chairs. The labeled data set is manually labeled after field collection. Set the number of training batches to 32 and the number of training rounds to 2000. Then start training the YOLOv7 network model. During the training process, monitor the various indicators of the model and continue training until the model converges, such as Figure 6 shown.

[0099] (2) After the model converges, use a variety of evaluation indicators and methods to evaluate the training effect, such as accuracy and recall, to determine whether the model meets the expected requirements. If the model's accuracy and other data do not meet the expected requirements, it is necessary to modify the training rounds, batches or data sets and retrain.

[0100] (3) Setting the parameters required by the Hungarian algorithm in the DeepSORT algorithm, including the maximum number of tracking frames, the number of skipped frames, and the distance threshold to the tracked target. If the drone is small and highly flexible, the distance threshold can be set relatively close to more accurately capture target features and information. The relevant threshold settings in this embodiment are shown in Table 1, which can prevent the drone from colliding with the target due to being too close, and use the equipment it carries to obtain a clear target image within an appropriate distance, thereby achieving effective tracking and monitoring.

[0101] (4) After training is completed, the weight hyperparameter file generated by the training is specified as the basis for YOLOv7 detection. Start the drone and connect the control terminal to the unmanned vehicle and the local area network created by the drone. This connection process needs to ensure the stability and security of the network to prevent data transmission interruption or interference.

[0102] (5) After the connection is successful, the control terminal controls the unmanned vehicle to carry the drone to the designated area. After arriving at the designated area, the control terminal controls the drone to take off. Before taking off, various safety checks must be performed on the drone, including whether the drone has sufficient power and whether the drone's propellers and motors are working properly.

[0103] (6) Specify the target to be tracked. The target number assignment principle is as follows: During the multi-target tracking process, the system assigns a unique number to each identified target. If the target is always within the field of view and remains matched, the number remains unchanged; once the target is lost and re-identified, or a new target is identified, the system will re-assign a number to it. The number value increases with the identification of new targets and will not be repeated. The control terminal provides a clear and concise operation interface (such as Figure 4 The operator enters the target number and clicks the "Start" button to start tracking.

[0104] ⑺ The drone completes the image recognition task with the help of the pre-trained YOLOv7 neural network model, and closely locks the target in combination with the tracking algorithm described in this article. In the actual tracking process, when the target moves, rotates, or is partially blocked, the drone continuously corrects its flight attitude and camera perspective through the dynamic adjustment capability of the tracking algorithm and the continuous target recognition feedback of the YOLOv7 model. After completing the task, the operator can click the "Land" button on the terminal interface, and the program will automatically perform a series of operations including finding the unmanned vehicle, tracking the unmanned vehicle, and landing on the unmanned vehicle.

Claims

1. A cross-domain collaborative tracking control method for UAVs and unmanned vehicles based on YOLOv7, characterized in that: The method comprises the following steps: S1: The user controls the unmanned vehicle to carry the drone to the designated location; S2: extract the frame images from the video stream collected by the drone in real time, and process the frame images to obtain the input image; S3: Input the input image into the pre-trained YOLOv7 network model to obtain all the target information in the input image, including the target type, confidence and the boundary of the target in the input image; S4: The user can specify target information and perform obstacle determination for targets not specified by the user; based on the target information in the input image, the path is updated in real time to avoid obstacles and continue to track the specified target; S5: The user issues a command to end tracking and land. The UAV searches for the target information of the unmanned vehicle in the input image, tracks and lands on the unmanned vehicle.

2. According to the YOLOv7-based cross-domain collaborative tracking control method for unmanned aerial vehicles and unmanned vehicles according to claim 1, it is characterized in that: In S2, the video stream collection and frame image processing include: the drone collects the video stream, discards some frame images to improve the processing speed, modifies the RGB color channels of the retained frame images to BGR to adapt to YOLOv7 image processing, and then reduces the aspect ratio of the frame images after the color channels are modified to 0.5 times the original to speed up the processing speed; the details are as follows: ⑴ Create a local area network: The drone creates a dedicated local area network and adds the terminal device to the local area network to ensure that the terminal device is successfully connected to the drone and use the data transmission protocol to obtain real-time video stream; ⑵Frame image preprocessing: Not all frame images in the acquired video stream are preprocessed; to improve the processing speed, the terminal will discard the frame images newly entering the buffer area when the frame image buffer area is accumulated or even overflows due to the slow processing speed: when the terminal's frame image buffer occupancy rate exceeds 50%, one frame is discarded for every four frames, when it exceeds 70%, one frame is discarded for every three frames, and when it exceeds 80%, one frame is discarded for every two frames; ⑶ Color channel modification: adjust the color channel of the retained frame image, only change the bit order of the color vector part, use the cvtColor function in the OpenCV library to implement, and adjust the encoding form to COLOR_RGB2BGR; (4) Image scaling: Reduce the frame image proportionally to 0.5 times of the original size, using the fast matrix multiplication in the opencv library. Specifically, use the resize function to set the size to 0.5 times the frame image.

3. According to the YOLOv7-based cross-domain collaborative tracking control method for unmanned aerial vehicles and unmanned vehicles according to claim 1, it is characterized in that: In S2, the pre-trained YOLOv7 network model is trained by the following steps: creating a data set containing category labels, a total of 1305 pictures; the data set contains label information of the unmanned vehicle equipped with a drone, the tracked object and obstacles; and training is performed by the following steps: ⑴ The training dataset is determined based on the specific usage scenario and tracking target. The characteristics of different scenarios and targets are significantly different. Indoor scenes are selected, and the tracking targets are people or unmanned vehicles. Ten types of target objects including stools, tables, chandeliers, people, and unmanned vehicles are collected, with a total of 1,305 labeled training data; the training data includes the target image and its corresponding bounding box and category label; (2) The training data covers data under special circumstances to improve the accuracy of model training; the more extensive the data included in the training data, the higher the accuracy of the model matching target; the training set includes data under various abnormal conditions, and the number of training times is set to 2000 to ensure the effective convergence of the model.

4. The cross-domain collaborative tracking control method of unmanned aerial vehicles and unmanned vehicles based on YOLOv7 according to claim 1 is characterized in that: In S4, (1) DeepSORT, a target tracking and recognition algorithm based on deep learning: The method for users to specify target information and lock targets uses DeepSORT, a target tracking and recognition algorithm based on deep learning, to achieve continuous tracking of multiple targets; the core of DeepSORT algorithm is Kalman filtering and Hungarian matching algorithm; The state of each target in the Kalman filter algorithm is represented by an eight-dimensional vector: Among them, u is the pixel horizontal coordinate of the geometric center point of the target in the image, v is the pixel vertical coordinate of the geometric center point of the target in the image, r is the width of the target boundary, and h is the height of the target boundary. is the pixel lateral velocity of the target, is the pixel longitudinal velocity of the target, is the speed of change of the target boundary width, is the speed of change of the target boundary height, and the speed value of each target state is initialized to 0; Kalman filtering is divided into two stages: (1) predicting the position of the target at the next moment, and (2) updating the target position based on the observation results. Assuming that the target motion state satisfies the uniform motion model: In formula (1) Indicates the motion state of the target at the previous moment. is the state transfer matrix, noise Assuming it is Gaussian white noise, its noise variance matrix is is a diagonal matrix, and the expression is as follows: Then the error covariance prediction is calculated as: At the same time, let the initial value of the state estimation error covariance be a diagonal matrix, and the initial value of the state estimation be the observed value, which have the following expressions: Build the observation model: Among them, H k is the observation transfer matrix, ν k is the observation noise, which represents the uncertainty of the observation quantity. Assume that ν k is Gaussian white noise, and its noise variance matrix is It maps the target state to the detection space, which has the following expression: Kalman filter gain update: Status update correction: Error covariance update correction: P′ k =P k -KH k P k (6) At this point, the iterative process of the Kalman filter is completed, and the position of the detection target in the image at the current moment is estimated; The motion and appearance information are integrated by combining two appropriate metrics; using the Mahalanobis distance between the predicted Kalman state and the newly arrived measurements: In formula (7), d (1) (i, j) is the Mahalanobis distance, d j is the target state currently measured and represented by an eight-dimensional vector, y i is the mean of the target state estimates, S i is the covariance matrix of the Kalman estimated state; the Mahalanobis distance takes into account the uncertainty of the state estimate by measuring how many standard deviations the detected position is from the mean trajectory; and the Mahalanobis distance is calculated by taking the 95% confidence interval from the inverse distribution χ 2 The Mahalanobis distance calculated in is used to exclude unlikely associations, and an indicator is used to indicate this decision: The second metric is integrated into the assignment problem; for each bounding box detection d j We obtain its global feature map through the CNN network and calculate its appearance descriptor r through l2 normalization j And ||r j ||=1; for each target’s predicted state, keep L k = 100 associated appearance descriptors, which represent the appearance features of the target in the first 100 frames: The minimum cosine distance between the i-th predicted state and the j-th measured target state in the appearance space is used as the second metric, which is defined as follows: Set a binary indicator for this metric: Cosine distance is less than or equal to threshold t (2) =0.6, it is assigned 1, otherwise it is assigned 0; by combining different aspects of the allocation problem, the two indicators complement each other; the weighted sum is used to combine the two indicators: c i，j =λd (1) (i,j)+(1-λ)d (2) (i,j) (12) When there is a large amount of displacement of the drone, the contribution of the target's motion information to the target tracking will be greatly reduced, but the target's appearance characteristics are not affected by the drone's motion, and λ = 0.1 is given; if the two indicators are within the gated area, it is called an acceptable association: ⑵ Mathematical model of UAV tracking control method: The above algorithm is used to obtain the accurate position information of the target in the input image, and use this information to adjust the flight attitude of the drone. The flight attitude includes: v(s) the forward speed of the drone and θ(d) the yaw angular velocity: Among them, v(s) is the forward speed of the UAV, s is the distance between the target and the UAV, which is obtained by using the optical flow method through visual information, v0 is the set starting speed, and s max is the maximum distance threshold, s min is the minimum distance threshold, and a is the increasing coefficient. It can be seen from formula (14) that as the target moves away from or approaches quickly, the speed of the drone will change linearly quickly to maintain an appropriate distance from the target. The yaw angle velocity model of the drone is introduced, and the yaw angle velocity model of the drone is as follows: e d =d-d mid in, is the yaw angular velocity, d, d safe ,d mid are the position of the target center in the input image, the safety threshold of the target center's deviation from the input image center, and the position of the input image center; a and b are increasing coefficients, e is d ′ is the deviation value at the previous moment; in formula (15), in order to make the change of the UAV heading smoother and avoid excessive turning amplitude when approaching the target center position, the deviation value measured at the previous moment is introduced into the current moment to produce a negative feedback effect; at the same time, the intensity of this negative feedback is limited by setting the coefficient; ⑶ Speed ​​model of drone obstacle avoidance control method: Independent judgment criteria and conditions are set up for the velocity vector and angular velocity vector of the drone. Specifically, for non-user-specified targets, computer vision is used to estimate the distance from the drone in three-dimensional space according to different target types and boundaries; a safety distance is set, and non-specified targets whose distance from the drone is greater than the safety distance are not considered; for each non-specified target whose distance is less than the safety distance, a three-dimensional space speed obstacle cone is established, and combined with the size of the drone itself, it is calculated whether the drone will collide with the non-specified target during tracking; In the face of possible collisions, in response to fast obstacles, the following velocity model is established: In formula (16), u(x) is the lateral velocity of the UAV, x, x safe are the lateral distance and lateral safety distance between the drone and the nearest obstacle in three-dimensional space respectively; the above model is applied to the vertical climbing speed of the drone: In formula (17), n(y) is the vertical velocity of the UAV, y, y sofe They are the vertical distance and vertical safety distance between the drone and the nearest obstacle in three-dimensional space respectively; The speed barrier cone in three dimensions is explained as follows: The speed obstacle VO generates a conical obstacle interval in the speed space of the drone. As long as the relative speed vector is outside the conical area, the drone will not collide with the obstacle in the future. In three-dimensional space, the radiation radius of the drone is R u , the radiation radius of a dynamic threat obstacle is R o ; The positions of the drone and the obstacle at a certain moment are P u , P o In the calculation, the position of the UAV is usually taken as the origin, and the position of the obstacle can be estimated from the visual information of the UAV; the velocity vector of the UAV is V u ; The obstacle velocity vector is V o ; UAV’s position information (P u , V u ) and the position information of obstacles (P o , V o ) can be expressed by formula (18) to formula (21): P u =(x u ,y u ,z u ) (18) P o =(x o ,y o ,z o ) (20) where v u 、v o are the velocity vector magnitudes of the drone and the obstacle respectively; θ u , and θ o , are the velocity vector direction angles of the drone and the obstacle, θ is the pitch angle, is the yaw angle; o As the center of the sphere, R is the radius to make a sphere with P u Make a cone and a sphere P as the vertex o Tangent; V uo is the relative vector velocity between the two, and the magnitude of the relative velocity is v uo , d is the Euclidean distance between the drone and the obstacle, and the relationship is as follows: R=R u +R o (26) V uo =V u -V o (27) If the vector V uo If the end is within the three-dimensional speed obstacle cone, the dynamic obstacle poses a threat to the drone. For these obstacles that pose a threat to the drone, the priority is divided according to the distance from the drone. The closer the distance, the higher the priority. The drone will avoid the obstacle with the highest priority and then repeat the avoidance process.

5. According to the YOLOv7-based cross-domain collaborative tracking control method for unmanned aerial vehicles and unmanned vehicles according to claim 1, it is characterized in that: In S5, the landing control method for tracking and landing on the unmanned vehicle is as follows: First, the unmanned vehicle is identified and detected by the YOLOv7 network model to obtain the target information of the unmanned vehicle. The position of the unmanned vehicle relative to the drone is estimated based on the pixel ratio of the unmanned vehicle target boundary in the input image and the preset unmanned vehicle distance ratio coefficient. Secondly, the drone controls the flight attitude to fly to the vicinity of the unmanned vehicle and turns on the downward camera of the drone. The video stream obtained by the downward camera is processed and input into the YOLOv7 network model. YOLOv7 recognizes the landing mark above the unmanned vehicle. Finally, the parallel PID control algorithm is used to control the speed of the drone, that is, to keep the relative position of the drone and the unmanned vehicle unchanged on the horizontal plane, and at the same time, the drone controls the altitude to gradually decrease until it lands on the unmanned vehicle. At this point, the drone can realize cross-domain collaborative tracking functions between drones and unmanned vehicles in dynamic and complex environments based on YOLOv7 and control algorithms.

Images