A multi-UAV relay tracking method

Abstract

The present invention relates to a multi-UAV relay tracking method, and belongs to the field of optoelectronic target tracking technology. The method comprises: UAVs patrol their respective working areas and obtain UAV videos; an airborne target detection model automatically detects the target of interest in the UAV video and obtains the pixel position and category of the target; according to the target detection result, the ground monitoring center selects the tracking target in the field of view of a certain UAV, and the airborne target tracking model continuously tracks the selected target, extracts the position and feature vector of the target, and shares it with neighboring UAVs; according to the target position, it is judged whether the target has entered the working area of ​​the neighboring UAV, and if so, the target detected by the neighboring UAV is re-identified according to the feature vector of the tracking target, and relay tracking is performed. The present invention realizes multi-UAV relay tracking by organically combining target detection, target tracking and target re-identification, and is suitable for security, patrol and other fields.

Description

Technical Field

[0001] The invention belongs to the technical field of photoelectric target tracking, and in particular relates to a multi-UAV relay tracking method. Background Art

[0002] Drones are flexible in deployment, highly maneuverable, and have a wide field of view. They are widely used in areas such as regional inspection, monitoring, and reconnaissance. Compared with other sensing methods, photoelectric drone target tracking has the advantages of low cost, strong adaptability, and high target clarity. At the same time, it does not require auxiliary means such as binding electronic tags or trackers to the target. It has been widely studied and used in recent years.

[0003] However, most of the methods and systems implemented in the existing technology are based on a single UAV platform. A single UAV is limited in flight time and communication radius and cannot achieve long-term continuous tracking of target flight. Therefore, how to use multiple UAVs to collaboratively track ground moving targets for a long time and accurately has important practical significance. Summary of the invention

[0004] In order to solve the limitations of the existing single UAV target tracking method, the present invention provides a multi-UAV relay tracking method, which uses multiple UAVs and an airborne target detection and tracking model to achieve long-term, stable and accurate tracking of ground moving targets.

[0005] The present invention is achieved through the following technical solutions:

[0006] A multi-UAV relay tracking method comprises the following steps:

[0007] Step 1, mission initialization: drones patrol their respective work areas and obtain drone videos;

[0008] Step 2, object detection: The onboard object detection model of each drone automatically detects the object of interest in the drone video and obtains the pixel position and category of the object;

[0009] Step 3, target tracking: Based on the target detection results, the ground monitoring center selects the tracking target in the field of view of a certain drone. The drone's onboard target tracking model continuously tracks the selected target, extracts the position and feature vector of the selected target, and shares it with neighboring drones.

[0010] Step 4, relay tracking: determine whether the selected target has entered the working area of ​​the neighboring drone based on the target position. If it has entered, re-identify the target detected by the neighboring drone based on the feature vector of the selected target and perform relay tracking.

[0011] Furthermore, the airborne target detection model is a YOLOv5-tiny lightweight target detection model, and the airborne target tracking model is a siamrpn tracking model.

[0012] Furthermore, the specific method of step 3 is:

[0013] Step 3.1: Based on the target detection results, the ground monitoring center selects a target of interest in the field of view of UAV A as the tracking target;

[0014] Step 3.2: The airborne target tracking model continues to track the selected target and extracts the target's bounding box, position, and feature vector F;

[0015] Step 3.3: The position of the selected tracking target is transmitted to the flight control module, and the UAV A is controlled to produce corresponding movements to ensure continuous tracking of the selected tracking target. At the same time, according to the bounding box of the selected tracking target and the attitude information of the payload, the payload is adjusted in real time to ensure that the selected tracking target is always in the middle of the UAV's field of view;

[0016] Step 3.4: While UAV A is continuously tracking the target, the target position and feature vector F of the selected tracking target are published to its neighboring UAVs in real time through the communication link;

[0017] Step 3.5: When the selected tracking target leaves the working area of ​​UAV A, stop tracking and switch to target detection mode.

[0018] Furthermore, the specific method of step 4 is:

[0019] Step 4.1: The neighboring UAV receives the target position and feature vector F sent by UAV A;

[0020] Step 4.2: According to the target position, adjust the posture of the neighboring UAV and the payload to ensure that the optical axis direction of the payload is at the target position;

[0021] Step 4.3: The onboard target detection model of the neighboring UAV automatically detects the target of interest in the field of view;

[0022] Step 4.4: If the tracked target has entered the working area of ​​the neighboring drone, for each target of interest detected in step 4.3, calculate the matching degree between it and the feature vector F of the tracked target. If the matching degree is greater than the threshold τ s If it is not found, it will be retained, otherwise it will be discarded, and finally all target sets that meet the conditions will be obtained;

[0023] Step 4.5: The airborne target tracking model of the neighboring UAV tracks all targets in the target set and extracts the feature vector F of each target i ';

[0024] Step 4.6: Calculate the feature vector F of each target in the target set i 'The matching degree with the feature vector F of the tracking target;

[0025] Step 4.7: Repeat steps 4.5 and 4.6 until the number of tracking frames reaches the specified number N. k ; For N in the target set s targets, and obtain a feature matching sequence:

[0026] Step 4.8: According to N k The frame target feature matching result determines whether each target in the target set meets the re-identification conditions from the two aspects of accuracy and stability:

[0027] Condition 1: For target i, the feature matching sequence contains at least B frame feature matching s i (k) is greater than the threshold τ h , where τ h >τ s , B≤N s ;

[0028] Condition 2: The stability of target i is calculated by counting the deviation ratio of feature matching. If the stability is greater than the threshold, target i is considered to meet condition 2. The calculation method of the stability of target i is as follows:

[0029] (1) Calculate the mean μ of the matching degree i and variance σ i :

[0030]

[0031]

[0032] (2) Calculate the feature matching deviation of each frame:

[0033] |s i (k)-μ i |

[0034] (3) The statistical feature matching deviation does not exceed 2σ i The number of frames N 2σ , and calculate the ratio This ratio is the stability of target i;

[0035] Only when target i satisfies both of the above conditions, target i is considered to meet the re-identification conditions;

[0036] Step 4.9: According to the target re-identification results, count the number of drones that can perform relay tracking Nuav , determine the relay tracking strategy:

[0037] N uav =0, that is, no drone can perform relay tracking. At this time, the control is exchanged to the monitoring center, waiting for the user to reselect the target;

[0038] N uav =1, that is, only one UAV can be relay tracked. At this time, the UAV is selected as the next UAV for relay tracking, and the target with the largest average feature matching degree is selected as the relay tracking target;

[0039] N r >1, that is, there are multiple UAVs that can perform relay tracking. At this time, the target with the largest average feature matching degree and the corresponding UAV are selected as the relay tracking target and the next UAV to perform relay tracking respectively.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] (1) Through the use of multiple UAVs and the onboard target detection and tracking model, long-term, stable, and accurate tracking of ground moving targets in a large area is achieved;

[0042] (2) For UAV relay tracking, a target re-identification strategy is proposed, which effectively improves the success rate of target relay tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a multi-UAV relay tracking method.

[0044] Figure 2 This is a schematic diagram of the multi-UAV patrol area.

[0045] Figure 3 It is a schematic diagram of a multi-UAV relay tracking system. DETAILED DESCRIPTION

[0046] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0047] like Figure 1As shown in the figure, a multi-UAV relay tracking method is provided. The UAVs patrol their respective working areas and obtain UAV videos. The airborne target detection model automatically detects the target of interest in the UAV video and obtains the pixel position and category of the target. According to the target detection result, the ground monitoring center selects the tracking target in the field of view of a certain UAV, and the airborne target tracking model continuously tracks the selected target, extracts the position and feature vector of the target, and shares them with neighboring UAVs. According to the target position, it is determined whether the target has entered the working area of ​​the neighboring UAV. If it has entered, the target detected by the neighboring UAV is re-identified according to the feature vector of the tracking target, and relay tracking is performed.

[0048] The method specifically comprises the following steps:

[0049] Step 1: Task initialization: According to the pre-set drone patrol area, each drone patrols its own work area and obtains drone video, including:

[0050] Step 1.1: Assume that there are 4 drones in total, and each drone corresponds to a different patrol area. The size and shape of the patrol area can be customized according to the mission requirements and the actual flight time and communication radius of the drone. Typically, the patrol area of ​​each drone can be a 5km×5km rectangular area, and each area has an overlapping area of ​​200m. The patrol areas of the four drones can form a large rectangle of 9.8km×9.8km, such as Figure 2 shown.

[0051] Step 1.2: According to the patrol plan, each drone takes off and begins patrolling its respective work area to obtain drone video.

[0052] Step 2: Target detection. Each drone performs target detection on the drone video frames by using the lightweight target detection model YOLOv5-tiny as the onboard target detector to obtain the target detection results, including the pixel position and category of the target.

[0053] Step 3: Target tracking. Based on the target detection results, the ground monitoring center selects the target of interest detected by a certain drone as the tracking object, uses the SIAMRPN tracking model as the airborne target tracker to continuously track the selected target, extracts the target's position and feature vector, and shares them with neighboring drones. Specifically, it includes:

[0054] Step 3.1: Based on the target detection results, the ground monitoring center selects the target of interest detected by UAV A in the patrol area R1 as the tracking object;

[0055] Step 3.2: Drone A uses the siamrpn tracking model as an onboard target tracker to continuously track the selected target and extract the target's bounding box B, position P, and feature vector F;

[0056] Step 3.3: The position P of the target is transmitted to the flight control module, and the UAV A is controlled to produce corresponding movement to ensure continuous tracking of the ground target. At the same time, according to the bounding box B of the target and the attitude information of the payload, the payload is adjusted in real time to ensure that the tracked target is always in the middle of the UAV's field of view;

[0057] Step 3.4: While UAV A is continuously tracking the target, it publishes the target position P and feature vector F to its neighboring UAVs in real time through the communication link;

[0058] It should be noted that the neighboring drones of a certain drone refer to other drones that have an intersection with the patrol area of ​​the drone. For example, the neighboring drones of drone A in the embodiment of the present invention are drones B, C, and D.

[0059] Step 3.5: When the target leaves the working area of ​​drone A, stop tracking the target and switch to target detection mode.

[0060] Step 4: Relay tracking: determine whether the target has entered the working area of ​​the neighboring drone based on the target position. If it has entered, re-identify the target detected by the neighboring drone based on the feature vector of the tracking target and perform relay tracking, which includes:

[0061] Step 4.1: The neighboring UAV receives the tracking target position P and feature vector F sent by UAV A;

[0062] Step 4.2: According to the target position, adjust the posture of the neighboring UAV and the payload to ensure that the optical axis direction of the payload is at the target position;

[0063] Step 4.3: The YOLOV5-tiny airborne target detection model of the neighboring drone automatically detects the target of interest in the field of view and extracts the feature vector F of the target i ';

[0064] It should be noted that F i ' represents the feature vector of the i-th target.

[0065] Step 4.4: If the target has entered the neighboring drone working area, calculate the target feature vector F according to each target detected in step 4.3 i 'The matching degree between the tracking target feature vector F, if the matching degree is greater than the specified threshold τ s If the target is not met, it will be retained, otherwise it will be discarded, and finally all the target sets that meet the conditions will be obtained. Where N s Indicates the number of targets that meet the conditions;

[0066] It should be noted that the feature vector matching degree is calculated using the cosine similarity calculation formula:

[0067]

[0068] Among them, F i '·F represents the eigenvector F i 'Inner product with the eigenvector F, ||F i '|| 2 Denotes the eigenvector F i ' modulus length, ||F|| 2 Represents the modulus of the eigenvector F.

[0069] Step 4.5: The airborne target tracking model of the neighboring UAV tracks all targets in the target set and extracts the feature vector F of each target in the current frame i ';

[0070] Step 4.6: Calculate the feature vector F of each target in the target set by cosine similarity i 'The matching degree with the tracking target feature vector F;

[0071] Step 4.7: Return to 4.5 and repeat the process until the number of tracking frames meets the specified number of frames N. k ;

[0072] It should be noted that for N in the target set s targets, we can get a target feature matching sequence:

[0073] Step 4.8: Calculate whether each target meets the target re-identification condition based on the matching degree sequence.

[0074] It should be noted that this method determines whether the target meets the re-identification conditions from two aspects: accuracy and stability:

[0075] Condition 1: For target i, the feature matching sequence contains at least B frame feature matching s i (k) is greater than the specified threshold τ h , where τ h >τ s , B≤N s ;

[0076] Condition 2: Stability is calculated by counting the deviation ratio of the target feature matching degree. The specific calculation method is as follows:

[0077] (1) For each target i, calculate the mean and variance of the matching degree:

[0078]

[0079]

[0080] (2) For each target i, calculate whether the feature matching deviation of each frame falls within 2σ i Range:

[0081] |s i (k)-μ i |≤2σ i

[0082] (3) For each target i, the statistical feature matching deviation falls within 2σ i The number of frames in the interval N 2σ , and calculate the ratio like If it is greater than a certain threshold, then target i is considered to meet condition 2.

[0083] Only when the target meets the above two conditions at the same time, it is said that the target meets the conditions for re-identification.

[0084] Step 4.9: According to the target re-identification results, count the number of drones that can perform relay tracking N uav , determine the relay tracking strategy.

[0085] It should be noted that the relay tracking capability is only available when the drone contains targets that meet the re-identification conditions.

[0086] If N uav =0, it means that there is no drone available for relay tracking, and the control needs to be exchanged to the monitoring center, waiting for the user to reselect the target.

[0087] If N uav = 1, select the drone that meets the conditions as the next drone for relay tracking, and select the drone with the largest average matching degree (i.e. N k The target with the average of the matching degrees is taken as the relay tracking target;

[0088] If N r >1, the target with the largest average matching degree and the corresponding UAV are selected as the relay tracking target and the next UAV to perform relay tracking respectively.

[0089] Figure 3 The multi-UAV relay tracking system for implementing the above method is shown, including the following modules:

[0090] (1) Airborne intelligent perception module, used to complete video acquisition and airborne target detection and tracking.

[0091] It should be noted that the airborne intelligent perception module includes four sub-modules, namely the gimbal camera, intelligent perception module, flight control module and communication module. The gimbal camera is responsible for acquiring the drone video image sequence and sending it to the intelligent perception module; the intelligent perception module is responsible for running the lightweight intelligent perception model on the embedded smart board to realize target detection, tracking and re-identification, and send the perception results to the communication module and flight control module; the flight control module is responsible for drone flight control, and adjusts the flight status according to the target perception results to ensure that the target is always within the drone's field of view; the communication module is responsible for communication between drones and between drones and ground monitoring modules. The communication content includes information such as the target's position, pixel coordinates and feature vector.

[0092] (2) Ground monitoring module, which is used to monitor the video images of each drone and the results of target detection and tracking in real time.

[0093] It should be noted that the ground monitoring module includes two sub-modules, namely the display module and the communication module. The display module is responsible for real-time display of the video images of each drone and the results of target detection and tracking; the communication module is responsible for the communication between each drone and the ground monitoring module.

[0094] The present invention realizes multi-UAV relay tracking through an organic combination of target detection, target tracking and target re-identification, and is suitable for fields such as security and patrol.

[0095] In summary, in view of the limitations of single UAV target tracking, the present invention proposes a multi-UAV relay tracking method and system, which utilizes intelligent perception models such as airborne target detection and tracking, and realizes long-term, stable and accurate tracking of ground targets of interest in a large range through the coordinated cooperation of multiple UAVs.

Claims

1. A multi-UAV relay tracking method, It is characterized in that The steps include: Step 1, mission initialization: drones patrol their respective work areas and obtain drone videos; Step 2, object detection: The onboard object detection model of each drone automatically detects the object of interest in the drone video and obtains the pixel position and category of the object; Step 3, target tracking: Based on the target detection results, the ground monitoring center selects the tracking target in the field of view of a certain drone. The drone's onboard target tracking model continuously tracks the selected target, extracts the position and feature vector of the selected target, and shares it with neighboring drones. Step 4, relay tracking: determine whether the selected target has entered the working area of ​​the neighboring drone based on the target position. If so, re-identify the target detected by the neighboring drone based on the feature vector of the selected target and perform relay tracking; During the re-identification process, the neighboring drones are selected to match the selected target with a degree greater than the threshold. The target of interest is obtained, and all targets in the target set are continuously tracked for a specified number of frames ; For the target set targets, and obtain a feature matching sequence: ; ;according to The frame target feature matching result determines whether each target in the target set meets the re-identification conditions from the two aspects of accuracy and stability: Condition 1: For the target , there are at least Feature matching of frames Greater than threshold ,in , ; Condition 2: Calculate the target by counting the deviation ratio of feature matching If the stability is greater than the threshold, the target Satisfy condition 2; target The stability is calculated as follows: (1) Calculate the mean of the matching degree and variance : (2) Calculate the feature matching deviation of each frame: (3) The statistical feature matching degree deviation does not exceed Number of frames , and calculate the ratio , this ratio is the target stability; Only when the target When both of the above conditions are met, the target Satisfy the re-identification conditions.

2. A multi-UAV relay tracking method according to claim 1, It is characterized in that The airborne target detection model is a YOLOv5-tiny lightweight target detection model, and the airborne target tracking model is a siamrpn tracking model.

3. A multi-UAV relay tracking method according to claim 1, It is characterized in that The specific method of step 3 is: Step 3.1: Based on the target detection results, the ground monitoring center selects a target of interest in the field of view of UAV A as the tracking target; Step 3.2: The airborne target tracking model continues to track the selected target and extracts the target's bounding box, position, and feature vector F; Step 3.3: The position of the selected tracking target is transmitted to the flight control module, and the UAV A is controlled to produce corresponding movements to ensure continuous tracking of the selected tracking target. At the same time, according to the bounding box of the selected tracking target and the attitude information of the payload, the payload is adjusted in real time to ensure that the selected tracking target is always in the middle of the UAV's field of view; Step 3.4: While UAV A is continuously tracking the target, the target position and feature vector F of the selected tracking target are published to its neighboring UAVs in real time through the communication link; Step 3.5: When the selected tracking target leaves the working area of ​​UAV A, stop tracking and switch to target detection mode.

4. A multi-UAV relay tracking method according to claim 3, It is characterized in that In step 4, the specific method of obtaining the feature matching degree sequence is: Step 4.1: The neighboring UAV receives the target position and feature vector F sent by UAV A; Step 4.2: According to the target position, adjust the posture of the neighboring UAV and the payload to ensure that the optical axis direction of the payload is at the target position; Step 4.3: The onboard target detection model of the neighboring UAV automatically detects the target of interest in the field of view; Step 4.4: If the tracked target has entered the working area of ​​the neighboring drone, for each target of interest detected in step 4.3, calculate the matching degree between it and the feature vector F of the tracked target. If the matching degree is greater than the threshold If it is not found, it will be retained, otherwise it will be discarded, and finally all target sets that meet the conditions will be obtained; Step 4.5: The airborne target tracking model of the neighboring UAV tracks all targets in the target set and extracts the feature vector of each target ; Step 4.6: Calculate the feature vector of each target in the target set and the feature vector of the tracking target The matching degree; Step 4.7: Repeat steps 4.5 and 4.6 until the number of tracking frames reaches the specified number of frames. ; For the target set targets, and obtain a feature matching sequence: ; .

5. A multi-UAV relay tracking method according to claim 3, It is characterized in that In step 4, the specific method of relay tracking is: According to the target re-identification results, count the number of drones that can perform relay tracking , determine the relay tracking strategy: ,That is, no drone is able to perform relay tracking, at which point the control is exchanged to the monitoring center, waiting for the user to reselect the target; , that is, only one UAV is capable of relay tracking. At this time, this UAV is selected as the next UAV for relay tracking, and the target with the largest average feature matching degree is selected as the relay tracking target; , that is, there are multiple UAVs capable of relay tracking. At this time, the target with the largest average feature matching degree and the corresponding UAV are selected as the relay tracking target and the next UAV to perform relay tracking, respectively.

Images