Multi-unmanned aerial vehicle target tracking method and system based on cross-view collaboration

Abstract

The invention discloses a multi-unmanned aerial vehicle target tracking method based on cross-view collaboration, which is suitable for robust target monitoring in a dynamic scene. The method comprises the following steps: firstly, establishing a spatial mapping relation among visual angles of multiple unmanned aerial vehicles through feature point matching and projection transformation; after each unmanned aerial vehicle node independently executes deep learning-based target detection, converting a detection frame center point coordinate of an auxiliary view angle into a main view angle coordinate system, and calculating an Euclidean distance between the detection frame center point coordinate and a main view angle detection result; a main visual angle potential leak detection area is identified by setting a dynamic distance threshold value, and a local high-sensitivity re-detection module is triggered based on leak detection confidence; and finally, realizing space-time tracking of observation data by adopting a single-view target tracking algorithm. According to the invention, the problem of leak detection caused by single-view-angle shielding is effectively solved through a cooperative verification mechanism of a cross-view-angle detection frame center point space relationship, and the continuity and reliability of target tracking in an unmanned aerial vehicle view angle complex environment are remarkably improved while the real-time performance is ensured.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a multi-UAV target tracking method and system based on cross-viewpoint collaboration. Background Art

[0002] In recent years, with the rapid development of drone technology and the increasing demand for multi-sensor collaborative operations, multi-drone systems have been increasingly used in fields such as national defense security, emergency rescue, traffic monitoring, and environmental monitoring. However, target tracking tasks in real-world application scenarios face complex dynamic changes and severe physical environment challenges, such as target occlusion, perspective changes, and distributed data fusion. Traditional tracking systems with a single perspective or fixed monitoring points are unable to cope with these complex situations. Traditional monitoring systems often rely on stationary cameras installed in fixed locations. Such systems mainly rely on a single perspective to acquire video data, and often use target detection algorithms based on convolutional neural networks (CNNs) (such as YOLO, Faster R-CNN, etc.) and tracking methods based on Kalman filters or correlation filters. However, the limitations of this single perspective are obvious in the following aspects:

[0003] Occlusion problem: When a target is partially occluded by a foreground object, a single perspective cannot provide enough information to accurately detect and identify the target, resulting in loss or misjudgment.

[0004] Blind spot coverage: Fixed camera viewing angles will create blind spots in surveillance. Even if multiple cameras are spliced ​​together, it is still difficult for the overall system to achieve global perception of target movement.

[0005] Continuous tracking of dynamic targets is difficult: Single-machine tracking algorithms (such as those based on Kalman filtering or simple association algorithms) often face problems of reduced detection accuracy and tracking interruption in multi-target scenarios due to insufficient local information in target-dense areas or fast-moving scenes. Summary of the Invention

[0006] In order to solve the problems of occlusion and poor blind spot information acquisition capabilities under the perspective of drones in the existing technology, the present invention proposes a multi-UAV target tracking method based on cross-perspective collaboration. By integrating detection information from multiple perspectives, the target tracking effect under each perspective is improved, effectively solving problems such as occlusion and blind spots.

[0007] According to one aspect of the present invention, a multi-UAV target tracking method based on cross-view collaboration is provided, comprising:

[0008] Match the images of the main view and the auxiliary view, and calculate the transformation matrix based on the matching results;

[0009] Perform target detection on the images of the main view and auxiliary view respectively to generate target detection results from different viewpoints;

[0010] Combined with the transformation matrix, the auxiliary view detection results are mapped to the main view and compared with the main view detection results. Based on the comparison results, the local area is re-detected to generate the final main view detection results;

[0011] Track the detection results of the final main view.

[0012] As a further technical solution, matching the images of the primary view and the auxiliary view includes:

[0013] Based on the LoFTR architecture, a detector-free image matching model based on local feature transformation residual optimization is constructed. This model inputs a shared convolutional feature extraction network for the main view and auxiliary view images, and extracts rich contextual features through a multi-scale feature fusion strategy. Subsequently, a context fusion module based on a gated recurrent unit is introduced to dynamically integrate multi-scale features.

[0014] As a further technical solution, the method further includes:

[0015] In the coarse matching stage, the Transformer structure introduced in the LoFTR architecture is used for local-global joint modeling, while adding a loss function based on epipolar geometry constraints;

[0016] In the fine matching stage, the differentiable RANSAC module is used to screen and eliminate the preliminary matching point set based on local motion consistency.

[0017] As a further technical solution, the detection results of the auxiliary view are mapped to the main view and compared with the detection results of the main view, and local area re-detection is performed based on the comparison results, including:

[0018] A collection constraint model for overlapping regions across views is established and integrated with an adaptive re-detection mechanism.

[0019] As a further technical solution, the detection results of the final main view are tracked, including:

[0020] Use the Tracking-by-detection paradigm to continuously track the final main view detection results.

[0021] As a further technical solution, the method further includes:

[0022] Receive the detection results of a single view and extract the appearance feature vector of each detection box through the re-identification feature extraction network;

[0023] Use the Kalman filter to predict the state of the next frame for the existing trajectory;

[0024] The detection data and prediction data of the current frame are matched with each other through the matching strategy, and the tracking results are updated through the trajectory processing strategy to output the tracking box information.

[0025] As a further technical solution, the method further includes:

[0026] The data association matching cost between frames fuses the appearance cost and the motion cost. The appearance cost is obtained by the re-identification feature extraction network, and the motion cost is obtained by the Kalman filter.

[0027] According to one aspect of the present invention, a multi-UAV target tracking system based on cross-view collaboration is provided, comprising:

[0028] Image matching module, used to match the images of the main view and the auxiliary view, and calculate the transformation matrix based on the matching results;

[0029] The target detection module is used to perform target detection on the images of the main view and the auxiliary view respectively, and generate target detection results of different viewpoints;

[0030] The cross-view enhancement module is used to map the auxiliary view detection results to the main view and compare them with the main view detection results by combining the transformation matrix. Based on the comparison results, the local area is re-detected to generate the final main view detection results.

[0031] The tracking module is used to track the detection results of the final main perspective.

[0032] According to one aspect of the present invention, a multi-UAV target tracking device based on cross-view collaboration is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the multi-UAV target tracking method based on cross-view collaboration.

[0033] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the multi-UAV target tracking method based on cross-view collaboration.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1) Through the collaborative design of image matching, target detection, cross-view enhancement and target tracking, the present invention significantly improves the accuracy and stability of target detection in complex scenarios such as occlusion interference and blind spots, and has good engineering adaptability and promotion value.

[0036] 2) During the target tracking process, the present invention introduces an enhanced trajectory management strategy. The system can continuously and smoothly predict and correct the target's motion trajectory based on the drone's perspective information, thereby enhancing the system's ability to capture small targets and targets with large changes in motion state from the drone's perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To facilitate understanding of the technical solutions of the present invention, the following accompanying drawings are provided. These drawings illustrate the structural design and workflow of the key modules of the present invention, providing a direct understanding of the innovative features and implementation methods of the present invention. It should be noted that these drawings are for illustrative purposes only, and the specific implementation details may be adjusted and optimized based on actual application scenarios.

[0038] Figure 1 A flowchart of a multi-UAV target tracking method based on cross-view collaboration is provided in an embodiment of the present invention.

[0039] Figure 2 This is a flowchart of image matching provided by an embodiment of the present invention.

[0040] Figure 3 A cross-view enhancement flow chart provided in an embodiment of the present invention.

[0041] Figure 4 This is a target tracking flowchart provided by an embodiment of the present invention.

[0042] Figure 5 This is a flow chart of trajectory management provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0044] like Figures 1 to 5 As shown, an embodiment of the present invention provides a multi-UAV target tracking method based on cross-view collaboration, comprising the following steps:

[0045] Step S01: Use an image matching model to match the images of the primary and auxiliary perspectives, and calculate the transformation matrix based on the matching results. First, an image matching method for cross-perspective target collaborative tracking is provided here, aiming to improve the image region registration accuracy under different perspectives.

[0046] In step S01, the image matching model is improved based on the LoFTR architecture to enhance the robustness and accuracy of feature matching in cross-view scenarios. To address the issue of insufficient feature consistency and prone to mismatching in traditional LoFTR under large viewpoint differences or occlusion, this embodiment proposes a context-aware feature consistency enhancement structure.

[0047] Specifically, the primary and auxiliary view images are first fed into a shared convolutional feature extraction network, where rich contextual features are extracted through a multi-scale feature fusion strategy. Subsequently, a contextual fusion module based on a gated recurrent unit (GRU) is introduced to dynamically integrate these multi-scale features. The gating weights of this module are adjusted by the view difference parameter between the images, adaptively enhancing feature consistency across viewpoints and thus improving the basic accuracy of coarse matching.

[0048] In the coarse matching stage, we rely on the Transformer structure introduced in LoFTR to perform local-global joint modeling, and at the same time add a loss function based on epipolar geometry constraints to restrict the physical rationality of matching point pairs from a geometric level, significantly suppressing the risk of mismatching caused by outliers.

[0049] Furthermore, a differentiable RANSAC module is used in the fine matching stage to filter and eliminate the initial matching point set based on local motion consistency. This module can effectively identify abnormal point pairs that do not conform to the continuous motion assumption and improve the overall matching accuracy while maintaining end-to-end trainability.

[0050] The specific process is as follows:

[0051] The preset coarse matching confidence threshold is T1, the upper limit of the fine matching iteration number is N1, and the tolerance for false matching is E. The main view image I1 and the auxiliary view image I2 are input into the shared visual feature encoder to extract the image feature maps F1 and F2. After that, a coarse matching module based on the LoFTR structure is used for preliminary matching to obtain the candidate matching point set M0, and to determine whether the number of matching point pairs meets the minimum pairing threshold N. minIf not, the low-confidence processing mechanism is triggered, prioritizing reducing T1 to expand the candidate set; if satisfied, the initial candidate matching point M0 is input into the context-aware feature fusion module to perform multi-scale context feature reconstruction on the feature maps F1 and F2. To achieve efficient fusion of cross-scale information, this module introduces a gated recurrent unit (GRU) structure, combining the perspective difference between the primary and auxiliary perspectives as a dynamic gating adjustment factor, thereby improving the consistency between features and matching accuracy.

[0052] Specifically, let the main view feature be F s , the auxiliary view feature is F a , the viewing angle difference between the two is Δ v =ViewDiff(Fs,Fa). s 、F a and Δ v The two are input into GRU together, and its update gate and reset gate are dynamically adjusted to generate the fusion feature representation h t The process can be expressed as:

[0053] z t =σ(W z ·[F s ,F a ]+U z ·h {t-1} +V z ·Δ v ) (1)

[0054] r t =σ(W r ·[F s ,F a ]+U r ·h {t-1} +V r ·Δ v ) (2)

[0055]

[0056] Where σ(·) represents the Sigmoid activation function, ⊙ represents element-wise multiplication, and W·, U·, and V· are learnable parameters. {t-1} Represents the hidden state (historical information) of the previous time step; r t Represents the reset gate (determines how much historical information to discard); z t Represents the update gate (determines how much new information vs. historical information to retain); Represents the candidate hidden state (temporary state before gating).

[0057] Then, the fused hidden state h tAfter further processing by the cross-scale fusion module φ(·), the consistency-enhanced fusion feature F is generated. fusion Finally, through the matching point generation function Ψ(·), the enhanced matching point set M1 is obtained: F fusion =φ(h t ), M1=Ψ(F fusion ).

[0058] According to the consistency of the matching direction and the trend of feature similarity changes, the number of reversals of the matching results is calculated in real time, and multiple reversal critical thresholds R1, R2, R3, and R4 are set. Whenever reversals reaches a certain critical value, the gate parameter gain of the coarse matching stage and the step parameter step of the fine matching stage are dynamically adjusted. Among them, gain is an exponential enhancement adjustment factor that controls the degree of compression of the matching candidate set; step is a linear step adjustment factor used to balance error control and feature sensitivity. When the consecutive Nsuccess matching points are stable (that is, the matching distance error is less than E), the Down operation is performed: the matching strategy is adjusted to

[0059]

[0060] When N appears fail When the secondary matching is unstable, perform the Up operation to avoid falling into the local optimum:

[0061] γ=γ×gain+step (6)

[0062] Afterwards, the enhanced matching points M1 are fed into the geometric consistency check module. Constraints are imposed on the point pairs in M1 based on the epipolar geometric error function, and points that do not satisfy epipolar consistency are removed. Furthermore, a differentiable RANSAC module is used to remove outliers with inconsistent motion directions, resulting in the final matching point set M_final. If the number of reversals does not reach the set threshold L (e.g., L = 12), the algorithm returns to step S01 and continues feature matching and adaptive adjustment. If the number of reversals reaches L, the iteration stops, and the stable image matching result M_final is output.

[0063] Calculated by feature descriptor and The similarity of the image is used to generate the initial matching point pair set, and the RANSAC is used to eliminate the mismatches that do not meet the geometric consistency and output M_final. and The transformation matrix T is obtained by least squares solution B→A :

[0064]

[0065] A·t=0,t=vec(T B→A )

[0066] T B→A =reshape(argmin ‖t‖＝1 ‖At‖2,3,3)

[0067] A is the coefficient matrix constructed from matching point pairs, used for least squares solution; t is the vectorized homography matrix parameter; vec(·) is a matrix vectorization operation, which stacks the matrix into a column vector by column; reshape(.) is a vector-to-matrix operation.

[0068] In step S02, the target detection model is used to perform target detection on the primary and auxiliary perspective images respectively, generating target detection results for different perspectives. In step S02, the input of the target detection model is a single-perspective video frame or image, and the output is the detection result in the video frame or image.

[0069] In this embodiment, the target detection network is implemented based on the YOLOv3 architecture, which performs efficient feature parsing on the input image through a single-stage detection framework. The network takes a 1920×1080 pixel RGB image as input and first extracts multi-scale features through a Darknet-53 backbone network containing 53 layers of convolution. The backbone uses a residual connection structure to optimize deep feature propagation. Subsequently, the network generates feature maps of three scales, 13×13, 26×26, and 52×52, through a feature pyramid fusion mechanism, corresponding to the detection requirements of large, medium, and small targets, respectively. Each detection head consists of three consecutive convolutional layers, in which the core convolution kernel is configured as a combination of 1×1, 3×3, and 1×1, and dynamic reorganization of spatial semantic information is achieved through channel compression and expansion.

[0070] At the target positioning level, the network presets 9 anchor boxes (generated based on COCO dataset clustering) and calculates the bounding box offset through the regression formula:

[0071]

[0072] Where (c x ,c y ) is the grid coordinate, (p ω ,p h ) is the anchor box benchmark size, and σ(·) is the coordinate offset range constrained by the Sigmoid function. x , t y represents the center offset of the bounding box predicted by the network, which is the predicted value output by the model; b x 、b y represents the calculated bounding box center coordinates; t w , t h Indicates the width and height of the anchor box; bw 、b h Represents the calculated bounding box width and height. Classification and confidence prediction are jointly optimized through binary cross entropy loss, and the total loss function integrates the coordinate error (CIoU) Confidence Error and classification error Three items:

[0073]

[0074] Represents the positioning loss calculated using CIoU (the error between the predicted bounding box and the real box in terms of center coordinates and width and height); represents the confidence error loss; Represents the classification error loss. coord ,λ obj ,λ class Represent the weights of the three error losses respectively.

[0075] The final output is filtered by non-maximum suppression (NMS) to generate a detection box with category label and confidence score.

[0076] In step S03, the cross-view enhancement module maps the auxiliary view detection results to the primary view, compares them with the primary view detection results, performs regional re-detection based on the results, and improves detection performance through a screening strategy. The cross-view enhancement module establishes a collective constraint model for cross-view overlapping areas and incorporates an adaptive re-detection mechanism.

[0077] After obtaining the detection frames of the main view and the auxiliary view, we process the detection frames to obtain the coordinates of the center points of the detection frames, as shown below:

[0078]

[0079] Among them, (x cen ,y cen ) represents the coordinates of the center point of the detection frame, (x min ,y min ) represents the coordinates of the lower left corner of the detection box, (x max ,y max ) represents the coordinates of the upper right corner of the detection box.

[0080] For the coordinates of the center point of the auxiliary view detection frame, we map it to the main view through the transformation matrix obtained in step S01.

[0081]

[0082] Among them, (x' cen ,y' cen) is the coordinate of the center point of the detection frame after transformation, T B→A is the transformation matrix from the auxiliary perspective to the main perspective.

[0083] Then we do the same processing on the image, mapping the image of the auxiliary view to the main view through the transformation matrix, and obtain the area overlapping with the main view. Based on this, we filter out the detection frames whose center points in the main view are in the overlapping area and the detection frames whose center points in the transformed auxiliary view are in the overlapping area.

[0084] On this basis, we match the transformed detection frame in the auxiliary view with the detection frame in the main view, and then obtain the detection frame in the auxiliary view that does not match the main view. Specifically, we determine whether the detection frames match by measuring the Euclidean distance between the center points. We sort the center points of the detection frames in the auxiliary view in the overlapping area in descending order according to the confidence level and traverse them. The point with the highest confidence level is M b Calculate the closest Euclidean distance between it and the center point of the detection frame in the overlapping area of ​​the main view, and the point with the closest Euclidean distance is M a Then the point M whose nearest Euclidean distance is less than the threshold th b Record it as the matching point and delete M b and the matching M a , and continue traversing until all the detection box center points in the auxiliary view in the overlapping area are traversed. Then we obtain the detection box center points in the auxiliary view in the overlapping area that do not match the main view detection results and select the 10 points with the highest confidence M ub , divide the original image of the main view into 4 blocks evenly, according to M ub The re-detection area is determined based on the location of

[0085] We created an enhanced version of the original dataset and used it to train the re-detection model. Specifically, we evenly split each image in the training and validation sets of the original dataset into four sub-images of equal size. We also proportionally split the object detection boxes in the original image and mapped them to their respective sub-images.

[0086] After obtaining the retested areas, we used the retest model to perform supplementary detection on the retested areas and integrated the filtered results into the final detection results.

[0087] D AE =D AE (score>ts) (12)

[0088] D AE =NMS(D AE ) (13)

[0089] IoU = IoU(D AE ,D A ) (14)

[0090] IoU max =max(IoU) (15)

[0091] D A =D A .app(D AE (IoU max <tr)) (16)

[0092] We detect the re-detection area and obtain the supplementary detection target set D AE Then we detect the target set D AE Perform a preliminary screening and remove the detection boxes whose confidence is lower than the set threshold ts. Then, use non-maximum suppression to further remove redundant detection results. Then, we calculate the detection result set D AE Compared with the original detection result set D in the main view A The intersection-over-union ratio between them is calculated and the maximum value is selected, as shown in formulas (14) and (15). max If the value is less than the set threshold tr, the detection frame of the supplementary detection area is added to the final detection result to obtain the final detection result of the main perspective.

[0093] In step S04, the target tracking model is used to track the detection results of the main view in multiple consecutive frames. The loss target tracking model uses the Tracking-by-detection paradigm to continuously track the detection results obtained in step S03.

[0094] In step S04, the detection results of a single view are received, and the apparent feature vector of each detection frame is extracted through the re-identification feature extraction network; the state of the next frame is predicted for the existing trajectory through the Kalman filter; the detection data and the predicted data of the current frame are matched by data association through the matching strategy, and finally the tracking result is updated through the trajectory processing strategy, and the tracking frame information is output.

[0095] In this embodiment, the strongsort algorithm is used as the basic target tracking algorithm, and the input is the image frame and the detection frame obtained in step S03.

[0096] First, the osnet feature extraction network is used to extract the re-identification feature f from each detection box. i t , re-identification features are obtained by exponential moving average (EMA) feature fusion Finally, the normalized cosine distance is used to obtain the appearance cost A a α represents the historical appearance state The weight of .

[0097]

[0098] In the motion cost calculation, ECC camera compensation is first performed on each target, and then state space modeling is performed to obtain an 8-dimensional state vector, where the prediction of the previous frame to the current frame is The detection of the current frame is z j , perform state prediction through NSA Kalman filter to obtain the next frame predicted position T and covariance matrix Σ of the existing trajectory i , according to the Mahalanobis distance calculation formula, the motion cost A is obtained m .

[0099]

[0100] Where p represents the parameters of the image distortion transformation (such as rotation and translation, used to describe the camera motion between adjacent frames); E ECC (p) represents the objective function for quantifying the image alignment effect (minimizing this value is used to find the optimal transformation parameters); ||·||: Euclidean norm; i r Indicates that the image area corresponding to the trajectory prediction position is intercepted from the reference frame (previous frame (t-1)); i w (p) represents the distorted image block obtained by applying the distortion parameter p to the image block of the candidate region of the current frame (t); represents the reference image block after zero-mean processing; Represents the distorted image patch after zero-mean processing.

[0101] Compared with the original Kalman filter, the NSA Kalman filter proposes an adaptive calculation of noise covariance The formula is:

[0102]

[0103] in, represents the adaptive measurement noise covariance at the kth moment, which is used to adjust the weight of the detection value in the Kalman filter; c k Represents the confidence score of the detection box at the kth moment; R k Represents the preset constant measurement noise covariance.

[0104] After obtaining the appearance cost and motion cost, vanilla matching is performed on the predicted data and the detection data through the joint cost, which serves as the basis for subsequent trajectory management.

[0105] C=λA a +(1-λ)A m (twenty two)

[0106] C represents the joint cost; λ is the weight factor that controls the ratio of appearance cost to motion cost.

[0107] The original Strongsort algorithm subsequently uses a simple one-stage IoU matching strategy to manage tracks. This embodiment implements the two-stage IoU matching strategy from the Bytetrack algorithm to manage tracks. Specifically, after vanilla matching, the matched detection data is divided into high-confidence targets D_high and low-confidence targets D_low based on a set threshold.

[0108] Then, a first-stage matching is performed, using the IoU distance to construct a similarity metric, and Hungarian matching to build a matching relationship. The D_high in the current frame is matched with the predicted box in the trajectory T. The successfully matched detection and trajectory pairs are used to update the Kalman filter and the trajectory status in the subsequent process. The unmatched detection boxes are placed in D_remain, and the unmatched trajectories are placed in T_remain.

[0109] The second stage of matching only matches the trajectories in T_remain with D_low. Successfully matched detection and trajectory pairs are also used to update the Kalman filter and trajectory status in the subsequent process. Trajectories that have not been successfully matched are stored in T_remain, and unmatched D_low are deleted.

[0110] Track management is then performed: Targets in T_remain are considered temporarily lost and transferred to T_lost. If they are not matched again for 30 consecutive frames, their tracks are deleted. For detected targets in D_remain, if the detection box score exceeds a set threshold and they have been detected for more than two consecutive frames, a new track is initialized. The new track and the matched track are fed into the Kalman filter for update, and the tracking box information is finally output.

[0111] The implementation of each embodiment of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a multi-UAV target tracking system based on cross-viewpoint collaboration. This system is used to implement a multi-UAV target tracking method based on cross-viewpoint collaboration described in the aforementioned method embodiment.

[0112] The system includes: an image matching module, which is used to match images of the main perspective and the auxiliary perspective, and calculate the conversion matrix based on the matching results; a target detection module, which is used to perform target detection on images of the main perspective and the auxiliary perspective respectively, and generate target detection results of different perspectives; a cross-perspective enhancement module, which is used to combine the conversion matrix, map the detection results of the auxiliary perspective to the main perspective and compare them with the detection results of the main perspective, perform local area re-detection based on the comparison results, and generate the final detection results of the main perspective; and a tracking module, which is used to track the generated final detection results of the main perspective.

[0113] An embodiment of the present invention provides a multi-UAV target tracking system based on cross-perspective collaboration. It addresses the problems of poor occlusion and blind spot information acquisition capabilities in the prior art from the perspective of UAVs. By adopting the aforementioned modules, it improves the target tracking effect from each perspective by integrating detection information from multiple perspectives, effectively solving problems such as occlusion and blind spots.

[0114] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0115] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention also provides a multi-UAV target tracking device based on cross-perspective collaboration, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the multi-UAV target tracking method based on cross-perspective collaboration.

[0116] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the multi-UAV target tracking method based on cross-perspective collaboration.

[0117] In summary, the present invention discloses a multi-UAV target tracking method based on cross-perspective collaboration, which is suitable for robust target monitoring in dynamic scenes. The method first establishes a spatial mapping relationship between multiple UAV perspectives through feature point matching and projection transformation; after each UAV node independently performs target detection based on deep learning, the coordinates of the center point of the auxiliary perspective detection frame are converted to the main perspective coordinate system, and the Euclidean distance between the auxiliary perspective detection frame and the main perspective detection result is calculated; by setting a dynamic distance threshold, the potential missed detection area of ​​the main perspective is identified, and the local high-sensitivity re-detection module is triggered based on the missed detection confidence; finally, a single-perspective target tracking algorithm is used to realize the spatiotemporal tracking of the observation data. The present invention effectively solves the missed detection problem caused by single-perspective occlusion through a collaborative verification mechanism of the spatial relationship of the center point of the cross-perspective detection frame, significantly improves the continuity and reliability of target tracking in complex environments of UAV perspectives while ensuring real-time performance, and can be applied to fields that require continuous target tracking, such as traffic monitoring and facility inspection.

[0118] The implementation of the technical solution of the present invention is not limited to a specific hardware configuration. Those skilled in the art can flexibly select software or hardware implementation methods based on actual needs. Based on the core concept of the present invention, any technical solution that adopts the method process described in the present invention and its equivalent transformations, including but not limited to embedded systems, cloud computing platforms, or application-specific integrated circuits, shall be considered within the scope of protection of the present invention.

[0119] The embodiments of the present invention are intended only to illustrate the feasibility of the technical solution. In actual applications, adjustments and optimizations may be made based on the needs of specific scenarios. Any reasonable modification, equivalent substitution, or technical extension based on the concept of the present invention, as long as it does not deviate from the technical principles and innovations proposed by the present invention, shall be included in the scope of protection of the claims of the present invention.

[0120] It should be noted that the method of the present invention can be implemented in a variety of computer-executable forms, including program code stored in various storage media, firmware programs on dedicated processing chips, etc. These various implementation forms are all natural extensions and applications of the technical solution of the present invention.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A multi-UAV target tracking method based on cross-view collaboration, characterized in that: include: Match the images of the main view and the auxiliary view, and calculate the transformation matrix based on the matching results; Perform target detection on the images of the main view and auxiliary view respectively to generate target detection results from different viewpoints; Combined with the transformation matrix, the auxiliary view detection results are mapped to the main view and compared with the main view detection results. Based on the comparison results, the local area is re-detected to generate the final main view detection results; Track the detection results of the final main view.

2. The multi-UAV target tracking method based on cross-view collaboration according to claim 1 is characterized in that: Matching the images of the main view and the auxiliary view, including: Based on the LoFTR architecture, a detector-free image matching model based on local feature transformation residual optimization is constructed. This model inputs a shared convolutional feature extraction network for the main view and auxiliary view images, and extracts rich contextual features through a multi-scale feature fusion strategy. Subsequently, a context fusion module based on a gated recurrent unit is introduced to dynamically integrate multi-scale features.

3. The multi-UAV target tracking method based on cross-view collaboration according to claim 2 is characterized in that: The method further comprises: In the coarse matching stage, the Transformer structure introduced in the LoFTR architecture is used for local-global joint modeling, while adding a loss function based on epipolar geometry constraints; In the fine matching stage, the differentiable RANSAC module is used to screen and eliminate the preliminary matching point set based on local motion consistency.

4. The multi-UAV target tracking method based on cross-view collaboration according to claim 1 is characterized in that: The auxiliary view detection results are mapped to the main view and compared with the main view detection results. Based on the comparison results, the local area is re-detected, including: A collection constraint model for overlapping regions across views is established and integrated with an adaptive re-detection mechanism.

5. The multi-UAV target tracking method based on cross-view collaboration according to claim 1 is characterized in that: Track the detection results of the final main view, including: Use the Tracking-by-detection paradigm to continuously track the final main view detection results.

6. The multi-UAV target tracking method based on cross-view collaboration according to claim 5 is characterized in that: The method further comprises: Receive the detection results of a single view and extract the appearance feature vector of each detection box through the re-identification feature extraction network; Use the Kalman filter to predict the state of the next frame for the existing trajectory; The detection data and prediction data of the current frame are matched with each other through the matching strategy, and the tracking results are updated through the trajectory processing strategy to output the tracking box information.

7. The multi-UAV target tracking method based on cross-view collaboration according to claim 6 is characterized in that: The method further comprises: The data association matching cost between frames fuses the appearance cost and the motion cost. The appearance cost is obtained by the re-identification feature extraction network, and the motion cost is obtained by the Kalman filter.

8. A multi-UAV target tracking system based on cross-view collaboration, characterized in that: include: Image matching module, used to match the images of the main view and the auxiliary view, and calculate the transformation matrix based on the matching results; The target detection module is used to perform target detection on the images of the main view and the auxiliary view respectively, and generate target detection results of different viewpoints; The cross-view enhancement module is used to map the auxiliary view detection results to the main view and compare them with the main view detection results by combining the transformation matrix. Based on the comparison results, the local area is re-detected to generate the final main view detection results. The tracking module is used to track the detection results of the final main perspective.

9. A multi-UAV target tracking device based on cross-view collaboration, characterized in that: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the multi-UAV target tracking method based on cross-view collaboration as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the multi-UAV target tracking method based on cross-view collaboration according to any one of claims 1 to 7.

Images