Fixed-wing cluster unmanned aerial vehicle target tracking and search and rescue method based on DeepSORT

Abstract

The invention discloses a DeepSORT-based fixed-wing cluster unmanned aerial vehicle target tracking and search-and-rescue method, which comprises the following steps that: unmanned aerial vehicles take off at the same time and enter a search-and-rescue area; each unmanned aerial vehicle detects a target in the image in real time to obtain a target detection result; a target is tracked and predicted through a DeepSORT algorithm, and track information of the target is obtained; each unmanned aerial vehicle transmits the position information and the corner mark of the target back to the surface search and rescue control computer; the search and rescue control computer fuses the target information returned by the plurality of unmanned aerial vehicles to obtain accurate target positions and corner marks; the search and rescue control computer distributes a search and rescue task to each unmanned aerial vehicle; each unmanned aerial vehicle executes search and rescue actions according to the distributed task, and updates the target position and corner mark information in real time through a positioning and communication module; and when the search and rescue task is completed or the time reaches the limit, the cluster unmanned aerial vehicle returns to the base. According to the invention, the cluster fixed-wing unmanned aerial vehicle can be effectively cooperated to execute a search and rescue task, and the flight capability and level of the fixed-wing cluster unmanned aerial vehicle are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a fixed-wing cluster unmanned aerial vehicle target tracking and search and rescue method based on DeepSORT. Background Art

[0002] After a natural disaster or accident, searching and rescuing people, materials, vehicles, equipment, etc. in a large area is very dangerous and difficult for rescuers and machinery to enter the scene due to the fire, disaster, and complex scenes of the target site. In traditional rescue methods, personnel need to enter the fire or disaster site for investigation and monitoring, facing dangers such as high temperature and thick smoke. The use of fixed-wing cluster drones can quickly fly over the fire scene, using high-definition cameras and thermal imaging technology to obtain real-time fire scene information, including the location of the fire source, the spread of the fire, etc., to provide reliable data support. During the search and rescue process, personnel may be limited by factors such as fire and smoke, making it difficult to accurately search and locate trapped people, and the movement of trapped people will bring greater resistance to the search and rescue. Fixed-wing cluster drones can quickly search the entire fire scene, and use the collaborative technology between drones to assign targets to different drones, quickly locate the location of trapped people and track the targets, and provide accurate target information. In the fire scene, the coordination and allocation of search and rescue tasks is a complex and difficult task. Traditional rescue methods require personnel to coordinate and allocate tasks, which faces the problems of time pressure and inaccurate information. However, using fixed-wing swarm drones, DeepSORT (Deep Simple Online and Realtime Tracking, a multi-target tracking algorithm based on deep learning) and collaboration between drones can quickly track and identify targets for search and rescue, and then assign each drone to the optimal search and rescue task through task allocation, thereby improving search and rescue efficiency and success rate. Therefore, using fixed-wing swarm drones for target search, tracking, fusion positioning and target allocation is necessary and efficient for rescue missions. Summary of the invention

[0003] In view of this, the present invention provides a fixed-wing swarm UAV target tracking and search and rescue method based on DeepSORT, which realizes search and rescue and task allocation with identification and tracking as the core through target allocation algorithm and UAV swarm collaboration, so as to achieve rapid response, reduce personnel risks and difficulties, improve rescue efficiency and success rate, etc., and can better cope with disaster scenes such as large-scale fires, mountain torrents and mudslides, ensure the safety of rescue personnel and provide effective rescue support.

[0004] The present invention discloses a fixed-wing cluster UAV target tracking and search and rescue method based on DeepSORT, which comprises: Step 1: The ground search and rescue control computer first sends the preset plan to each drone in the fixed-wing cluster, so that the drones take off at the same time and enter the search and rescue area; Step 2: Each drone detects the target in the image in real time through the Yolov5 target detection model to obtain the target detection result; Step 3: Based on the target detection results, the target is tracked and predicted through the DeepSORT algorithm to obtain the target trajectory information; Step 4: Each UAV transmits the target's location information and corner mark back to the surface search and rescue control computer through the positioning and communication module; Step 5: The search and rescue control computer integrates the target information sent back by multiple drones to obtain the accurate target position and corner mark; Step 6: The search and rescue control computer assigns the search and rescue mission to each UAV based on the accurate target location and corner mark; each UAV performs the search and rescue operation according to the assigned mission, and updates the target location and corner mark information in real time through the positioning and communication module; when the search and rescue mission is completed or the time limit is reached, the cluster UAV returns to the base.

[0005] Furthermore, the step 2 comprises: Each drone uses the Yolov5 target detection model for target matching. The Yolov5 target detection model is loaded using PyTorch. The weight file is loaded into the Yolov5 target detection model. The Yolov5 target detection model is set to evaluation mode to eliminate gradient updates. The image acquired by the drone is input into the Yolov5 target detection model. Through the forward propagation process, the Yolov5 target detection model generates the bounding box coordinates, category label and confidence score of the target. Among them, the bounding box indicates the position of the target in the image, the category label indicates the category to which the target belongs, and the confidence score indicates the model's confidence in the target. The target type is determined by the category label, and the target type is determined according to the index or name of the category label.

[0006] Furthermore, the step 3 comprises: Step 31: Create a corresponding tracker according to the detection result of the first frame; initialize the motion variable of the Kalman filter, and predict the adjacent frames corresponding to the first frame through the Kalman filter; Step 32: Perform IOU matching on the frame coordinate corner points of the target detected in the current frame and the frame coordinate corner points predicted by the tracker in the previous frame, and calculate the cost matrix based on the IOU matching results; the cost matrix reflects the matching cost between each target detection frame and each tracking frame; Step 33: Using the cost matrix as the input of the Hungarian algorithm to obtain a first matching result; the first matching result is a cost matrix obtained by calculating the Mahalanobis distance and the cosine distance between the target detection box and the tracking box; Step 34: Repeat steps 32 and 33 until a tracker with a determined state appears or the video frame ends; Step 35: Use the Kalman filter to predict the boxes corresponding to the tracker in the determined state and the tracker in the undetermined state; perform cascade matching of the motion information and the identified target on the tracker in the determined state and the detected target; Step 36: After cascade matching, a second matching result is obtained; the second matching result is that the tracker matches or the identified target does not match the tracker; when the second matching result is that the tracker matches, the tracker updates its corresponding tracker variable through Kalman filtering; when the second matching result is that the identified target does not match the tracker, the tracker in the previously undetermined state and the unmatched tracker are matched one by one with the unmatched detected target, and then the IOU matching result is used; and the cost matrix is ​​calculated; Step 37: All cost matrices obtained in step 36 are used as inputs of the Hungarian algorithm to obtain linear matching results; Step 38: Repeat steps 35 to 37 until the current frame is matched with the adjacent frame.

[0007] Further, the first matching result is a tracker mismatch, a recognition mismatch, or a successful pairing of the detection frame and the prediction frame; When the tracker does not match, if the tracker is in an uncertain state, the unmatched tracker will be deleted; if it is in a determined state, it will be deleted only after reaching the specified survival time continuously; When the recognition does not match, the recognition target is initialized as a new track; The successful pairing of the detection frame and the prediction frame indicates that the previous frame and the next frame have been successfully tracked, and the corresponding tracker variables are updated through Kalman filtering according to the corresponding detection results.

[0008] Furthermore, when the linear matching result is that the tracker does not match, the unmatched tracker is directly deleted; when the linear matching result is that the unmatched target is identified, the identified target is initialized as a new tracker; when the linear matching result is that the current detection frame and the predicted frame are successfully paired, it indicates that the previous frame and the next frame are tracked; if the previous frame and the next frame are tracked successfully, the identified target is updated to its corresponding tracker variable through Kalman filtering.

[0009] Further, the step 32 comprises: Step 321: extracting the detection frame and tracking frame of each target in the target detection result by DeepSORT algorithm; calculating the Mahalanobis distance between the detection frame and tracking frame of the target; Step 322: The detection frame and the tracking frame of the target are normalized to a hypersphere of a unit ball through L2, and the cosine distance between the detection frame and the tracking frame of the target is calculated; an element of the cost matrix is ​​obtained according to the calculated Mahalanobis distance and cosine distance; Step 323: Repeat steps 321 and 322 until all targets in the target detection results are traversed, and then output the cost matrix.

[0010] Further, the step 33 comprises: Step 331: All elements in each row of the cost matrix are subtracted from the minimum element in the row; all elements in each column of the cost matrix are subtracted from the minimum element in the column; Step 332: Use the least horizontal or vertical lines to cover all zero elements in the cost matrix; Step 333: If the number of lines is equal to the specified value, the optimal allocation is found and the algorithm ends, otherwise it goes to step 334; Step 334: Find the minimum element in the cost matrix that is not covered by any line, subtract the minimum element from each row that is not covered by a line, and add the minimum element to each column that is covered by a line; Step 335: Return to step 332, and continue to use the least horizontal or vertical lines to cover all zero elements in the cost matrix until the optimal allocation is found, and finally output the optimal matching matrix. Each element in the optimal matching matrix corresponds to an element in the cost matrix, indicating the best matching relationship between the target detection box and the tracking box.

[0011] Furthermore, the step 5 comprises: Obtain the position coordinates and confidence level of each target collected by the search and rescue control computer; Calculate the mean and standard deviation of the confidence, use them to calculate the weight of each target, and perform weighted fusion; Normalize the weight of each target; multiply the position coordinates of each target by its corresponding normalized weight, add up all the multiplied coordinates, and output the fused coordinates.

[0012] Furthermore, the step 6 comprises: Determine the number of target points and drones, initialize the task list of each drone to be empty, calculate the distance between all drones and each target point in the search and rescue area, sort all the distances obtained, and obtain a distance list; select the drone corresponding to the smallest distance in the distance list; then determine whether to assign the target point to the drone. If the drone is not in a mission state or an abnormal state, add the target point to the drone's task list, jump out of the current loop, and continue to process the next target point; otherwise, continue to determine whether to assign the target point to the next drone until all target points are assigned and a task list corresponding to each drone is obtained.

[0013] Due to the adoption of the above technical solution, the present invention has the following advantages: 1. The present invention is reasonably designed, efficient, and stable, has considerable generalization ability and high intelligence level, can effectively coordinate cluster fixed-wing UAVs to perform search and rescue missions, and improve the flight capability and level of fixed-wing cluster UAVs.

[0014] 2. Through the target allocation algorithm and the coordinated use of drone clusters, search and rescue with identification and tracking as the core and task allocation are realized. It has the advantages of rapid response, reduced personnel risks and difficulties, improved rescue efficiency and success rate, and solves the problems of traditional search and rescue that easily cause casualties, coordination difficulties, and long time periods. It can better deal with disaster scenes such as large-scale fires, mountain torrents and mudslides, ensure the safety of rescuers, and provide effective rescue support. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 It is a schematic diagram of the interaction between the mission scenario and the ground-to-air command and rescue framework of an embodiment of the present invention; Figure 2 A schematic diagram of DeepSORT framework tracking logic according to an embodiment of the present invention; Figure 3 A schematic diagram of fusing similar targets observed by multiple drones through information fusion according to an embodiment of the present invention; Figure 4 The present invention is a flowchart of a fixed-wing swarm UAV target tracking and search and rescue method based on DeepSORT according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0018] The present invention provides an embodiment of a fixed-wing cluster UAV target tracking and search and rescue method based on DeepSORT, which comprises the following steps: Step 1: The ground search and rescue control computer first sends the preset plan to each drone in the fixed-wing cluster, so that the drones take off at the same time and enter the search and rescue area; Step 2: Each drone detects the target in the image in real time through the Yolov5 target detection model to obtain the target detection result; Step 3: Based on the target detection results, the target is tracked and predicted through the DeepSORT algorithm to obtain the target trajectory information; Step 4: Each UAV transmits the target's location information and corner mark back to the surface search and rescue control computer through the positioning and communication module; Step 5: The search and rescue control computer integrates the target information sent back by multiple drones to obtain the accurate target position and corner mark; Step 6: The search and rescue control computer assigns the search and rescue mission to each UAV based on the accurate target location and corner mark; each UAV performs the search and rescue operation according to the assigned mission, and updates the target location and corner mark information in real time through the positioning and communication module; when the search and rescue mission is completed or the time limit is reached, the cluster UAV returns to the base.

[0019] In this embodiment, step 2 includes: Each drone uses the Yolov5 target detection model for target matching. The Yolov5 target detection model is loaded using PyTorch. The weight file is loaded into the Yolov5 target detection model. The Yolov5 target detection model is set to evaluation mode to eliminate gradient updates. The image acquired by the drone is input into the Yolov5 target detection model. Through the forward propagation process, the Yolov5 target detection model generates the bounding box coordinates, category label and confidence score of the target. Among them, the bounding box indicates the position of the target in the image, the category label indicates the category to which the target belongs, and the confidence score indicates the model's confidence in the target. The target type is determined by the category label, and the target type is determined according to the index or name of the category label.

[0020] In this embodiment, step 3 includes: Step 31: Create a corresponding tracker according to the detection result of the first frame; initialize the motion variable of the Kalman filter, and predict the adjacent frames corresponding to the first frame through the Kalman filter; Step 32: Perform IOU matching on the frame coordinate corner points of the target detected in the current frame and the frame coordinate corner points predicted by the tracker in the previous frame, and calculate the cost matrix based on the IOU matching results; the cost matrix reflects the matching cost between each target detection frame and each tracking frame; Step 33: Using the cost matrix as the input of the Hungarian algorithm to obtain a first matching result; the first matching result is a cost matrix obtained by calculating the Mahalanobis distance and the cosine distance between the target detection box and the tracking box; Step 34: Repeat steps 32 and 33 until a tracker with a determined state appears or the video frame ends; Step 35: Use the Kalman filter to predict the boxes corresponding to the tracker in the determined state and the tracker in the undetermined state; perform cascade matching of the motion information and the identified target on the tracker in the determined state and the detected target; Step 36: After cascade matching, a second matching result is obtained; the second matching result is that the tracker matches or the identified target does not match the tracker; when the second matching result is that the tracker matches, the tracker updates its corresponding tracker variable through Kalman filtering; when the second matching result is that the identified target does not match the tracker, the tracker in the previously undetermined state and the unmatched tracker are matched one by one with the unmatched detected target, and then the IOU matching result is used; and the cost matrix is ​​calculated; Step 37: All cost matrices obtained in step 36 are used as inputs of the Hungarian algorithm to obtain linear matching results; Step 38: Repeat steps 35 to 37 until the current frame is matched with the adjacent frame.

[0021] In this embodiment, the first matching result is that the tracker does not match, the recognition does not match, or the detection frame and the prediction frame are paired successfully; When the tracker does not match, if the tracker is in an uncertain state, the unmatched tracker will be deleted; if it is in a determined state, it will be deleted only after reaching the specified survival time continuously; When the recognition does not match, the recognition target is initialized as a new track; The successful pairing of the detection frame and the prediction frame indicates that the previous frame and the next frame have been successfully tracked, and the corresponding tracker variables are updated through Kalman filtering according to the corresponding detection results.

[0022] In this embodiment, when the linear matching result is that the tracker does not match, the unmatched tracker is directly deleted; when the linear matching result is that the unmatched target is identified, the identified target is initialized as a new tracker; when the linear matching result is that the current detection frame and the predicted frame are successfully paired, it indicates that the previous frame and the next frame are tracked; if the previous frame and the next frame are tracked successfully, the identified target is updated to its corresponding tracker variable through Kalman filtering.

[0023] In this embodiment, step 32 includes: Step 321: extracting the detection frame and tracking frame of each target in the target detection result by DeepSORT algorithm; calculating the Mahalanobis distance between the detection frame and tracking frame of the target; Step 322: The detection frame and the tracking frame of the target are normalized to a hypersphere of a unit ball through L2, and the cosine distance between the detection frame and the tracking frame of the target is calculated; an element of the cost matrix is ​​obtained according to the calculated Mahalanobis distance and cosine distance; Step 323: Repeat steps 321 and 322 until all targets in the target detection results are traversed, and then output the cost matrix.

[0024] In this embodiment, step 33 includes: Step 331: All elements in each row of the cost matrix are subtracted from the minimum element in the row; all elements in each column of the cost matrix are subtracted from the minimum element in the column; Step 332: Use the least horizontal or vertical lines to cover all zero elements in the cost matrix; Step 333: If the number of lines is equal to the specified value, the optimal allocation is found and the algorithm ends, otherwise it goes to step 334; Step 334: Find the minimum element in the cost matrix that is not covered by any line, subtract the minimum element from each row that is not covered by a line, and add the minimum element to each column that is covered by a line; Step 335: Return to step 332, continue to use the least horizontal or vertical lines to cover all 0 elements in the cost matrix until the optimal allocation is found, and finally output the optimal matching matrix, each element in the optimal matching matrix corresponds to an element in the cost matrix, indicating the best matching relationship between the target detection box and the tracking box. For example, if the element (i, j) in the optimal matching matrix indicates that the target detection box i matches the tracking box j, then this matching relationship is obtained by minimizing the element (i, j) in the cost matrix.

[0025] In this embodiment, step 5 includes: Obtain the position coordinates and confidence level of each target collected by the search and rescue control computer; Calculate the mean and standard deviation of the confidence, use them to calculate the weight of each target, and perform weighted fusion; Normalize the weight of each target; multiply the position coordinates of each target by its corresponding normalized weight, add up all the multiplied coordinates, and output the fused coordinates.

[0026] In this embodiment, step 6 includes: Determine the number of target points and drones, initialize the task list of each drone to be empty, calculate the distance between all drones and each target point in the search and rescue area, sort all the distances obtained, and obtain a distance list; select the drone corresponding to the smallest distance in the distance list; then determine whether to assign the target point to the drone. If the drone is not in a mission state or an abnormal state, add the target point to the drone's task list, jump out of the current loop, and continue to process the next target point; otherwise, continue to determine whether to assign the target point to the next drone until all target points are assigned and a task list corresponding to each drone is obtained.

[0027] For ease of understanding, the present application provides a more specific embodiment: like Figures 1 to 4 As shown in the figure, the ground search and rescue control computer first sends the preset plan to each fixed-wing cluster drone, so that the cluster fixed-wing drones take off at the same time and enter the search and rescue area. After entering the area, each drone detects the target object in the image in real time through the Yolov5 target detection model, and transmits the detection result to the DeepSORT target tracking algorithm. The DeepSORT algorithm tracks and predicts the target to obtain the trajectory information of the target. Each drone returns the target position information and corner mark to the search and rescue control computer through the positioning and communication module. The search and rescue control computer uses the target fusion algorithm to fuse the target information returned by multiple drones to obtain the accurate target position and corner mark. The search and rescue control computer uses a greedy algorithm to assign the search and rescue task to each drone according to the target position and corner mark information. Each drone performs the search and rescue operation according to the assigned task, and updates the target position and corner mark information in real time through the positioning and communication module.

[0028] When the search and rescue mission is completed or the time limit is reached, the swarm drones return to the base.

[0029] In one embodiment of the present application, the target identification and detection process of each drone is as follows: A pre-trained Yolov5 model is installed and configured on each drone. This model is able to detect target objects in images in real time. The model uses a pre-trained Yolov5 model for matching. The pre-trained Yolov5 model is loaded using PyTorch, the weight file is loaded into the model, the model is set to evaluation mode to eliminate gradient updates, and the pre-processed camera image is input into the loaded Yolov5 model. Through the forward propagation process, the pre-trained Yolov5 model will generate the bounding box coordinates, category label, and confidence score of the target, where the bounding box indicates the location of the target in the image, the category label indicates the category to which the target belongs, and the confidence score indicates the model's confidence in the target.

[0030] Each drone can then run object detection locally, and the category label can be used to determine which target type it belongs to. By using the labels of the COCO dataset, which contains 80 different target categories, the target type can be determined based on the index or name of the category label.

[0031] In one embodiment of the present application, Figure 2 As shown, the logic of access target tracking after identification is as follows: (1) Create a corresponding tracker based on the detection results of the first frame. Initialize the motion variables of the Kalman filter and predict the corresponding frame through the Kalman filter. At this time, the tracker must be in an unconfirmed state.

[0032] (2) Perform IOU matching on the frame coordinate corner points detected by the target in this frame and the frame coordinate corner points predicted by the tracker in the previous frame, and then calculate the cost matrix based on the IOU matching results; the cost matrix reflects the matching cost between each target detection box and each tracking box.

[0033] (3) The cost matrix obtained in the previous step is used as the input of the Hungarian algorithm to obtain the matching result; the matching result is the cost matrix obtained by calculating the Mahalanobis distance and cosine distance between the target detection box and the tracking box. At this time, three results are obtained. The first is that the tracker does not match. At this time, the tracker is in an uncertain state, so the unmatched tracker is deleted. If it is in a certain state, it must reach a certain survival time continuously before it can be deleted; the second is to identify mismatches and initialize such identified targets as new tracking; the third is that the detection frame and the prediction frame are paired successfully, which means that the previous frame and the next frame have been successfully tracked, and the corresponding recognition results update their corresponding tracker variables through the Kalman filter.

[0034] (4) Repeat steps (2) to (3) until a confirmed tracker appears or the video frame ends.

[0035] (5) Use Kalman filtering to predict the corresponding boxes of the confirmed tracker and the unconfirmed tracker. Perform cascade matching between the confirmed tracker frame and the identified target (each time the tracker is matched, the appearance features and motion information of the identified target are saved. By default, the first 100 frames are saved, using the appearance features and motion information) and the motion information of the identified target. This is done because the confirmed tracker is more likely to match the identified target).

[0036] (6) There are three possible results after cascade matching. The first is tracker matching, where the tracker updates its corresponding tracker variables through Kalman filtering. The second and third methods are that the identified target does not match the tracker. At this time, the previously unconfirmed trackers and unmatched trackers are matched with the unmatched identified targets one by one, and then the IOU matching results are used to calculate their cost matrix.

[0037] (7) All cost matrices obtained in (6) are used as the input of the Hungarian algorithm to obtain the linear matching results. At this point, three results are obtained. The first is that the tracker does not match. The unmatched tracker is directly deleted; the second is to identify the unmatched target and initialize such an identified target as a new tracker. The third is to successfully pair the detection frame and the prediction frame, which means that the previous frame and the next frame are tracked. If successful, the corresponding identified target is updated to its corresponding tracker variable through the Kalman filter.

[0038] (8) Repeat steps (5)-(7) until the frame ends.

[0039] Regarding (2), performing IOU matching on the frame coordinate corner points detected by the target in the current frame and the frame coordinate corner points predicted by the tracker in the previous frame, and then calculating the cost matrix based on the IOU matching results, the specific steps include: Step (21): extract the detection frame and tracking frame of each target in the target detection result by using the DeepSORT algorithm; calculate the Mahalanobis distance between the detection frame and the tracking frame of the target; Step (22): The detection frame and the tracking frame of the target are L2 regularized onto a hypersphere of a unit ball, and the cosine distance between the detection frame and the tracking frame of the target is calculated; an element of the cost matrix is ​​obtained according to the calculated Mahalanobis distance and cosine distance; Step (23): Repeat steps (21) and (22) until all targets in the target detection results are traversed, and then output the cost matrix.

[0040] Regarding (3), the cost matrix obtained in the previous step is used as the input of the Hungarian algorithm to obtain the matching result, which specifically includes the following steps: Step (31): All elements in each row of the cost matrix are subtracted from the minimum element of the row; all elements in each column of the cost matrix are subtracted from the minimum element of the column; Step (32): Use the least number of horizontal or vertical lines to cover all zero elements in the cost matrix; Step (33): If the number of lines is equal to the specified value, the optimal allocation is found and the algorithm ends, otherwise it goes to step (34); Step (34): Find the minimum element in the cost matrix that is not covered by any line, subtract the minimum element from each row that is not covered by a line, and add the minimum element to each column that is covered by a line; Step (35): Return to step (32) and continue to use the least horizontal or vertical lines to cover all zero elements in the cost matrix until the optimal allocation is found. Finally, the optimal matching matrix is ​​output. Each element in the optimal matching matrix corresponds to an element in the cost matrix, indicating the best matching relationship between the target detection box and the tracking box.

[0041] In one embodiment of the present application, after Yolov5 performs target detection, it will output the type and confidence of each target. According to each detected target, DeepSORT extracts its feature vector to represent the target. That is, a convolutional neural network is used to encode the target image area into a feature vector of fixed length. CNN(x) is used to represent the multiplication and addition operation of the CNN on the input tensor x. out Represents the output feature vector, and input represents the input image area tensor, which is calculated as follows:

[0042] DeepSORT then calculates the distance between each tensor through Mahalanobis distance and cosine distance to form a cost matrix, uses the Hungarian algorithm to process the cost matrix, associates the detection results of the current frame with the targets of the previous frame, and performs IOU matching. The algorithm determines the matching relationship between targets by minimizing the distance between them. The distance is calculated based on the feature vector of the target.

[0043] The Mahalanobis distance is expressed as The calculation is as follows:

[0044] in, Indicates the position of the jth detection frame; represents the predicted position of the target by the i-th tracker; S represents the covariance matrix between the detection box and the tracking box. and To find the difference, as mentioned earlier, due to the correlation and order of magnitude, it is necessary to use the covariance matrix S to normalize it, and finally obtain the Mahalanobis distance between the i-th tracking and the i-th detection.

[0045] The cosine distance is then calculated as follows:

[0046] The cosine distance uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals, and calculates the angle between the vectors through the cosine theorem to determine whether the directions of the two vectors are consistent.

[0047] By inputting the eigenvector mentioned above, it is normalized to a hypersphere of a unit ball through L2 regularization. Find the cosine distance between two eigenvectors, Corresponding to the jth detected feature vector, The corresponding tracked feature vector, where the tracked feature vector is a set represented by R i , retain the features of the past k successful tracking times.

[0048] By weight , so the matrix value of a pair of elements i, j in the cost matrix is for:

[0049] Then the calculation steps are as follows using the Hungarian algorithm: Step A. Row minimum element subtraction: All elements in each row of the cost matrix are subtracted from the minimum element of the row. The purpose of this step is to make at least one element in each row become 0, thereby simplifying the subsequent matching process.

[0050] Column minimum element subtraction: All elements in each column of the cost matrix are subtracted from the minimum element of the column. The purpose of this step is to make at least one element in each column become 0, thereby further simplifying the matching process.

[0051] Step B. Cover all 0 elements: Use the least horizontal or vertical lines to cover all 0 elements in the cost matrix. The purpose of this step is to determine which 0 elements are feasible matches.

[0052] Step C. Check the number of covered lines: If the number of lines is equal to the specified value (usually the dimension of the matrix), the optimal allocation is found and the algorithm ends. Otherwise, go to step D.

[0053] Step D. Adjust the cost matrix: Find the minimum element in the cost matrix that is not covered by any line, subtract the minimum element from each row that is not covered by a line, and add the minimum element to each column that is covered by a line. The purpose of this step is to introduce new 0 elements to increase the possibility of matching.

[0054] Step E. Return to step B and continue to use the least horizontal or vertical lines to cover all 0 elements in the cost matrix until the optimal allocation is found.

[0055] Through target association, DeepSORT is able to build the trajectory of the target. Each target's trajectory contains a series of positions and other attributes. During the tracking process, DeepSORT updates the state of the target and predicts the position of the target in the next frame by using a Kalman filter.

[0056] The Kalman filter prediction calculation is as follows:

[0057] Where x is the mean value of the tracker at time t-1, F is called the state transfer matrix, and this formula predicts x' at time t.

[0058]

[0059] Where P is the covariance of the tracker at time t-1, Q is the noise matrix of the system, representing the reliability of the entire system, and is initialized to a very small value. This formula predicts the .

[0060] Based on the target's trajectory history and prediction model, DeepSORT can predict the target's location in future frames. This prediction helps track the target more accurately and provides a certain degree of error tolerance when the target is lost or occluded.

[0061] In one embodiment of the present application, Figure 3 As shown in the figure, after the ground search and rescue control computer obtains the observation situation of all UAVs on the target, the algorithm process of obtaining the target fusion information is as follows: Collect the location coordinates and confidence of each target. Assuming there are n targets, the location coordinates of each target are expressed as , the confidence level is expressed as , where i=1,2,...,n.

[0062] For each target's confidence, a Gaussian function is used to convert it into a weight. The Gaussian weight is calculated using the following formula:

[0063] in, represents the weight of the i-th target, represents the confidence of the i-th target, μ represents the mean of the Gaussian distribution, and σ represents the standard deviation of the Gaussian distribution. The mean μ is set to the average value of the confidence, and the standard deviation σ is set to the standard deviation of the confidence.

[0064] The calculated weights are then normalized to ensure that the sum of the weights is 1. The normalization method is to divide each weight by the sum of all weights.

[0065]

[0066] in, Represents the normalized weight.

[0067] The target positions are weighted and fused according to the normalized weights. For each target position coordinate, multiply it by the corresponding weight, and then add all the weighted coordinates to get the output fusion coordinates.

[0068] The fused coordinates are (x out ,y out ), whose value is calculated as:

[0069] Through the above process, the position coordinates and confidence levels of n targets can be weightedly fused to obtain the output fused coordinates.

[0070] In one embodiment of the present application, the search and rescue control computer on the ground performs task allocation through a greedy algorithm. First, the search and rescue control computer collects information of the clustered fixed-wing drones (including the locations of the drones) and the accurate target locations and corner marks.

[0071] The search and rescue priority is determined based on the classification results of the identified targets. Targets with high priority should be assigned to drones first, and drones that are closer to the target are more likely to be assigned tasks. This is used to create a target allocation evaluation function to assign tasks to each target. For each target, select the drone that best meets the task allocation rules for assignment. A greedy algorithm is used to select the best drone each time to complete the search and rescue mission of the current target. The process is as follows: A1. Determine the number of target points and drones, and initialize the task list of each drone to be empty.

[0072] B1 For each target point, calculate its distance to all drones and sort the distances from small to large.

[0073] C1. For each target point, select the closest drone in order of distance.

[0074] D1. Assign the current target point to the UAV and update the UAV's task list.

[0075] E1. Repeat steps C1 and D1 until all target points are assigned or the task lists of all drones are full.

[0076] F1. If there are still unallocated target points, it means that the number of drones is insufficient. Consider increasing the number of drones or readjusting the allocation strategy.

[0077] G1. Output the task list of each drone, that is, get the final allocation plan.

[0078] After the task is assigned, the corresponding drone status and target status are updated. The drone that has been assigned a task is marked and its status is updated to busy. The target that has been assigned a task is marked and its status is updated to searched and rescued.

[0079] The task assignment and status update are repeated until all targets have been searched and rescued or the drones cannot be assigned tasks.

[0080] Specifically, the pseudo code of the task allocation algorithm is shown in Table 1:

[0081] Assume there are four targets {x1,x2,x3,x4} and five drones {drone 1, drone 2, drone 3, drone 4, drone5} performs the target allocation task as follows: enter: Target point list: A list X={x1,x2,x3,x4} containing the target points to be assigned.

[0082] Drone list: Contains a list of available drones D={drone1,drone2,drone3,drone4,drone5}.

[0083] Output: The task list of each UAV: ​​Contains the task list {task1,task2,task3,task4,task5} of the target point assigned to each UAV.

[0084] process: Initialize each drone's task list to be empty. {task1,task2,task3,task4,task5}={0,0,0,0,0}.

[0085] For each target point, calculate the distance between the drone and the target point, and sort them in order from small to large to get DIST_LIST= .

[0086] For each target point, iterate through the list of drones and select the one with the smallest distance.

[0087] For the currently traversed drone, determine whether the target point can be assigned to the drone, that is, the drone is not in a mission state or an abnormal state. If it can be assigned, add the target point to the drone's task list, jump out of the current loop, and continue to process the next target point. If it cannot be assigned to the current drone, continue to judge the next drone.

[0088] After all target points are assigned, the task list of each UAV is output.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fixed-wing swarm UAV target tracking and search and rescue method based on DeepSORT, characterized in that: include: Step 1: The ground search and rescue control computer first sends the preset plan to each drone in the fixed-wing cluster, so that the drones take off at the same time and enter the search and rescue area; Step 2: Each drone detects the target in the image in real time through the Yolov5 target detection model to obtain the target detection result; Step 3: Based on the target detection results, the target is tracked and predicted through the DeepSORT algorithm to obtain the target trajectory information; Step 4: Each UAV transmits the target's location information and corner mark back to the surface search and rescue control computer through the positioning and communication module; Step 5: The search and rescue control computer integrates the target information sent back by multiple drones to obtain the accurate target position and corner mark; Step 6: The search and rescue control computer assigns the search and rescue mission to each UAV based on the accurate target location and corner mark; each UAV performs the search and rescue operation according to the assigned mission, and updates the target location and corner mark information in real time through the positioning and communication module; when the search and rescue mission is completed or the time limit is reached, the cluster UAV returns to the base.

2. The method according to claim 1, characterized in that: The step 2 comprises: Each drone uses the Yolov5 target detection model for target matching. The Yolov5 target detection model is loaded using PyTorch. The weight file is loaded into the Yolov5 target detection model. The Yolov5 target detection model is set to evaluation mode to eliminate gradient updates. The image acquired by the drone is input into the Yolov5 target detection model. Through the forward propagation process, the Yolov5 target detection model generates the bounding box coordinates, category label and confidence score of the target. Among them, the bounding box indicates the position of the target in the image, the category label indicates the category to which the target belongs, and the confidence score indicates the model's confidence in the target. The target type is determined by the category label, and the target type is determined according to the index or name of the category label.

3. The method according to claim 1, characterized in that The step 3 comprises: Step 31: Create a corresponding tracker according to the detection result of the first frame; initialize the motion variable of the Kalman filter, and predict the adjacent frames corresponding to the first frame through the Kalman filter; Step 32: Perform IOU matching on the frame coordinate corner points of the target detected in the current frame and the frame coordinate corner points predicted by the tracker in the previous frame, and calculate the cost matrix based on the IOU matching results; the cost matrix reflects the matching cost between each target detection frame and each tracking frame; Step 33: Using the cost matrix as the input of the Hungarian algorithm to obtain a first matching result; the first matching result is a cost matrix obtained by calculating the Mahalanobis distance and the cosine distance between the target detection box and the tracking box; Step 34: Repeat steps 32 and 33 until a tracker with a determined state appears or the video frame ends; Step 35: Use the Kalman filter to predict the boxes corresponding to the tracker in the determined state and the tracker in the undetermined state; perform cascade matching of the motion information and the identified target on the tracker in the determined state and the detected target; Step 36: After cascade matching, a second matching result is obtained; the second matching result is that the tracker matches or the identified target does not match the tracker; when the second matching result is that the tracker matches, the tracker updates its corresponding tracker variable through Kalman filtering; when the second matching result is that the identified target does not match the tracker, the tracker in the previously undetermined state and the unmatched tracker are matched one by one with the unmatched detected target, and then the IOU matching result is used; and the cost matrix is ​​calculated; Step 37: All cost matrices obtained in step 36 are used as inputs of the Hungarian algorithm to obtain linear matching results; Step 38: Repeat steps 35 to 37 until the current frame is matched with the adjacent frame.

4. The method according to claim 3, characterized in that The first matching result is a tracker mismatch, a recognition mismatch, or a successful pairing of a detection frame and a prediction frame; When the tracker does not match, if the tracker is in an uncertain state, the unmatched tracker will be deleted; if it is in a determined state, it will be deleted only after reaching the specified survival time continuously; When the recognition does not match, the recognition target is initialized as a new track; The successful pairing of the detection frame and the prediction frame indicates that the previous frame and the next frame have been successfully tracked, and the corresponding tracker variables are updated through Kalman filtering according to the corresponding detection results.

5. The method according to claim 3, characterized in that: When the linear matching result is that the tracker does not match, the unmatched tracker is directly deleted; when the linear matching result is that the unmatched target is identified, the identified target is initialized as a new tracker; when the linear matching result is that the current detection frame and the predicted frame are successfully paired, it indicates that the previous frame and the next frame are tracked; if the previous frame and the next frame are tracked successfully, the identified target is updated to its corresponding tracker variable through Kalman filtering.

6. The method according to claim 3, characterized in that: The step 32 comprises: Step 321: extracting the detection frame and tracking frame of each target in the target detection result by DeepSORT algorithm; calculating the Mahalanobis distance between the detection frame and tracking frame of the target; Step 322: The detection frame and the tracking frame of the target are normalized to a hypersphere of a unit ball through L2, and the cosine distance between the detection frame and the tracking frame of the target is calculated; an element of the cost matrix is ​​obtained according to the calculated Mahalanobis distance and cosine distance; Step 323: Repeat steps 321 and 322 until all targets in the target detection results are traversed, and then output the cost matrix.

7. The method according to claim 3, characterized in that The step 33 comprises: Step 331: All elements in each row of the cost matrix are subtracted from the minimum element in the row; all elements in each column of the cost matrix are subtracted from the minimum element in the column; Step 332: Use the least horizontal or vertical lines to cover all zero elements in the cost matrix; Step 333: If the number of lines is equal to the specified value, the optimal allocation is found and the algorithm ends, otherwise it goes to step 334; Step 334: Find the minimum element in the cost matrix that is not covered by any line, subtract the minimum element from each row that is not covered by a line, and add the minimum element to each column that is covered by a line; Step 335: Return to step 332, and continue to use the least horizontal or vertical lines to cover all zero elements in the cost matrix until the optimal allocation is found, and finally output the optimal matching matrix. Each element in the optimal matching matrix corresponds to an element in the cost matrix, indicating the best matching relationship between the target detection box and the tracking box.

8. The method according to claim 1, characterized in that: The step 5 comprises: Obtain the position coordinates and confidence level of each target collected by the search and rescue control computer; Calculate the mean and standard deviation of confidence, use them to calculate the weight of each target, and perform weighted fusion; Normalize the weight of each target; multiply the position coordinates of each target by its corresponding normalized weight, add up all the multiplied coordinates, and output the fused coordinates.

9. The method according to claim 1, characterized in that: The step 6 comprises: Determine the number of target points and drones, initialize the task list of each drone to be empty, calculate the distance between all drones and each target point in the search and rescue area, sort all the distances obtained, and obtain a distance list; select the drone corresponding to the smallest distance in the distance list; then determine whether to assign the target point to the drone. If the drone is not in a mission state or an abnormal state, add the target point to the drone's task list, jump out of the current loop, and continue to process the next target point; otherwise, continue to determine whether to assign the target point to the next drone until all target points are assigned and a task list corresponding to each drone is obtained.

Images