Enhanced unmanned aerial vehicle-to-unmanned aerial vehicle tracking method, electronic equipment and storage medium

Abstract

The invention provides an enhanced unmanned aerial vehicle-to-unmanned aerial vehicle tracking method, electronic equipment and a storage medium, and the method comprises the following steps: obtaining image sequence data in a target unmanned aerial vehicle tracking scene, inputting the obtained image sequence data into a preset integrated model, extracting the feature information of the target unmanned aerial vehicle, and obtaining the feature information of the target unmanned aerial vehicle; the integrated model generates a response diagram according to the feature information of the target unmanned aerial vehicle; the integrated model evaluates the reliability of a current tracking result based on a space-time consistency principle in combination with response diagram analysis and trajectory prediction, and constructs a candidate target set according to the evaluation of the reliability of the tracking result; according to the integrated model, an OSTrack uniflow basic tracker, a target surrounding matching mechanism unit and a Transform-based trajectory prediction module are fused, response diagram analysis and trajectory prediction are combined, a tracking result is evaluated by using a space-time consistency principle, a candidate target set is constructed, dependence on confidence score is reduced, interference is effectively eliminated, and tracking reliability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to an enhanced unmanned aerial vehicle tracking method, an electronic device and a storage medium. Background Art

[0002] With the advancement of drone technology, drones have been widely used in various industries. Due to their flexibility and easy control, drones can be used to track and monitor target drones and capture or suppress them with the help of additional equipment. With the development of deep learning technology, deep neural networks have been widely used in visual object tracking in recent years. These networks can build strong appearance features through offline end-to-end training and accurately track targets using regression heads. Early deep trackers mainly relied on convolutional neural networks (CNNs) based on Siamese architecture, which significantly improved the performance compared with traditional tracking algorithms such as correlation filters.

[0003] For example, Zhang Hongwei, Li Xiaoxia, Zhu Bin, and Zhang Yang (Two-stage target tracking method based on twin neural network, 2021, 50(09): 341-352) proposed a two-stage target tracking method based on twin network to solve the problem that deep learning technology cannot eliminate the interference of similar targets. Compared with the one-stage SiamRPN twin network, the method in this paper better solves the problem that the original algorithm cannot take into account both generalization ability and anti-interference ability. The correlation filter modulation plus two target box regression makes the model have better accuracy.

[0004] Although this algorithm performs well in natural scenes, it overly relies on the confidence score of the network. In highly dynamic UAV-to-UAV tracking scenarios, the confidence score of the appearance model may become unreliable, resulting in tracking failure. In addition, current tracking algorithms often lack effective utilization of target trajectory information, or they rely on simple trajectory prediction methods such as Kalman filters, which are not suitable for highly dynamic scenes. This deficiency makes it difficult to handle the complex task of UAV-to-UAV tracking. To this end, an enhanced UAV-to-UAV tracking method is proposed. Summary of the invention

[0005] In view of this, an embodiment of the present invention hopes to provide an enhanced UAV-to-UAV tracking method to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide an enhanced UAV-to-UAV tracking method, comprising the following steps: obtaining image sequence data in a target UAV tracking scene, and inputting the obtained image sequence data into a preset integrated model to extract feature information of the target UAV, wherein the integrated model generates a response graph according to the feature information of the target UAV; the integrated model is based on the principle of spatiotemporal consistency, combined with response graph analysis and trajectory prediction, to evaluate the reliability of the current tracking result, and to construct a candidate target set according to the evaluation of the reliability of the tracking result; the consistency score is calculated for the trajectory sequence generated by each candidate target in the candidate target set, and the candidate target with the highest score is selected, and its initial position is used as the target position; the trajectory sequence corresponding to the target position is reversed, the trajectory is reversely predicted, and the mean square error between the predicted position and the actual position of each reverse predicted trajectory on each frame is calculated; the candidate target with the smallest mean square error is selected as the real target, thereby obtaining the final tracking result to improve the accuracy and reliability of UAV-to-UAV tracking.

[0007] As a further preferred embodiment of the present technical solution: the integrated model includes: an OSTrack single-stream basic tracker for processing input image sequence data; a target surrounding matching mechanism unit for further analyzing the image sequence data processed by the OSTrack single-stream basic tracker; and a trajectory prediction module for predicting the trajectory of the target UAV, wherein the trajectory prediction module predicts the trajectory of the UAV based on Transformer.

[0008] As a further preferred embodiment of the present technical solution: the image sequence data in the target UAV tracking scene is obtained, and the obtained image sequence data is input into a preset integrated model to extract the feature information of the target UAV, and the integrated model generates a response graph according to the feature information of the target UAV, including: obtaining the image sequence data in the target UAV tracking scene according to a predetermined sampling frequency by an image acquisition device carried on the tracking UAV, and the image sequence data is used to show the visual information of the target UAV and its surrounding environment; inputting the obtained image sequence data into the integrated model, and the OSTrack single-stream basic tracker in the integrated model performs feature extraction operations on the template area and the search area in the image sequence data by using the self-attention mechanism; wherein the template area is selected as a specific image area containing the target UAV, and the search area is set to a larger range area where the target UAV may exist in the current frame; generating a response graph according to the extracted feature information, and the feature information of the target UAV includes but is not limited to color features, texture features, shape features, and relative position features of the target UAV and the surrounding environment.

[0009] As a further preferred embodiment of the present technical solution: the response map analysis includes: setting a threshold to identify positions in the response map with scores higher than the threshold, obtaining candidate target bounding boxes based on a soft non-maximum suppression algorithm, and determining the tracking state according to the number of candidate boxes; at the same time, the OSTrack single-stream basic tracker predicts the target position according to the obtained candidate target bounding boxes.

[0010] As a further preferred embodiment of the present technical solution: the tracking status is determined according to the number of candidate boxes, including: when the number of candidate boxes is one, the reliability of the current tracking result is determined to be high and the tracking state is stable; when the number of candidate boxes is greater than one, the reliability of the current tracking result is determined to be low, and all targets corresponding to the candidate boxes are included in the candidate target set.

[0011] As a further preferred embodiment of the present technical solution: the trajectory prediction includes: using the Transformer-based trajectory prediction module in the integrated model to obtain the trajectory sequence of the target UAV in the previous multiple frames; the trajectory prediction module predicts the future trajectory of the target UAV in the next frame based on the acquired trajectory sequence of the target UAV in the previous multiple frames; comparing the predicted position of the trajectory prediction module with the target position predicted by the OSTrack single-stream basic tracker, and after calculating the Euclidean distance, comparing the calculated result with a preset distance threshold; if the calculated result is not higher than the preset distance threshold, the predicted position of the trajectory prediction module is determined to be normal, otherwise it is determined to be abnormal, and the predicted result is included in the candidate target set as a candidate target.

[0012] As a further preferred embodiment of the present technical solution: the calculation of the consistency score of the trajectory sequence generated by each candidate target in the candidate target set, selecting the candidate target with the highest score, and taking its initial position as the target position, comprises: determining that each candidate target generates a trajectory sequence ; The consistency score is calculated using the following formula :

[0013]

[0014] in, is the candidate target selection unit, Indicates Candidates in frame The location of , , represents the total number of candidate targets, is the total length of the time series, indicating the total number of frames from the first frame to the last frame, is the upper limit of the sum, that is, the total number of frames considered when calculating the consistency score; where the candidate position is evaluated and the expected true target position The candidate with the highest score is determined as the most likely target, and the initial position corresponding to the candidate is used as the position of the target.

[0015] As a further preferred embodiment of the present technical solution: the step of reversing the trajectory sequence corresponding to the target position, reversely predicting the trajectory, and calculating the mean square error between the predicted position and the actual position of each reversely predicted trajectory in each frame includes:

[0016] The trajectory sequence corresponding to the determined target position is reversely arranged in time order, so that the last frame of the original trajectory sequence becomes the starting frame, and the starting frame becomes the last frame;

[0017] The reversed trajectory sequence is input into the trajectory prediction module based on Transformer. The trajectory prediction module predicts the possible position of the target UAV in each reverse frame by weighted fusion of information at different positions based on the spatiotemporal correlation features between frames in the trajectory sequence.

[0018] For each reverse prediction trajectory, obtain its predicted position on each frame, and at the same time, extract the actual position information of the target drone on the corresponding frame from the original image sequence data;

[0019] According to the mean square error formula, the mean square error between the predicted position and the actual position of each of the reverse prediction trajectories in each frame is calculated, and the mean square error is used to show the degree of deviation between the reverse prediction trajectory and the actual trajectory.

[0020] To solve the above technical problems, another technical solution adopted in the present application is: a computer device, which includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the steps of an enhanced drone-to-drone tracking method as described above.

[0021] In order to solve the above technical problems, another technical solution adopted in the present application is: a storage medium storing program instructions capable of implementing an enhanced drone-to-drone tracking method as described above.

[0022] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0023] The integrated model of the present invention integrates the OSTrack single-stream basic tracker, the target surrounding matching mechanism unit and the Transformer-based trajectory prediction module, combines response graph analysis with trajectory prediction, uses the principle of spatiotemporal consistency to evaluate tracking results, constructs a candidate target set, reduces dependence on confidence scores, effectively eliminates interference, and improves tracking reliability.

[0024] The present invention is based on the Transformer trajectory prediction module, which can accurately predict the next frame position of the target drone by learning the previous multi-frame trajectory sequence. Even in the case of temporary invisibility or high-dynamic scenes, its position can be accurately estimated, reducing the risk of tracking failure and enhancing the stability and accuracy of tracking.

[0025] The present invention calculates the consistency score to screen the target position, performs reverse prediction and mean square error calculation on the trajectory sequence to determine the real target, and uses multi-threaded parallel processing to track the same target in the forward and backward directions. While ensuring the tracking accuracy, the real-time performance of the system is effectively improved to meet the actual needs of drone-to-drone tracking.

[0026] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 A schematic diagram of an enhanced framework proposed for an embodiment of the method of the present invention;

[0030] Figure 3 It is a schematic diagram of testing when the method of the present invention is actually applied;

[0031] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0033] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0034] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0035] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0037] Figure 1 FIG. 1 is a flow chart of an enhanced UAV-to-UAV tracking method according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present application is not limited to the method of FIG. Figure 1 The process sequence shown is limited. Figure 1-2 As shown: An enhanced UAV to UAV tracking method, comprising the following steps:

[0038] Step S100: acquiring image sequence data in a target UAV tracking scene, and inputting the acquired image sequence data into a preset integrated model to extract feature information of the target UAV, wherein the integrated model generates a response graph according to the feature information of the target UAV;

[0039] Among them, the integrated model includes: an OSTrack single-stream basic tracker for processing input image sequence data; a target surrounding matching mechanism unit for further analyzing the image sequence data processed by the OSTrack single-stream basic tracker; a trajectory prediction module for predicting the trajectory of the target drone, and the trajectory prediction module predicts the drone trajectory based on Transformer. In the present invention, OSTrack is selected as the single-stream basic tracker, and OSTrack is integrated with the target surrounding matching mechanism and the Transformer-based trajectory prediction module to ensure robust and accurate tracking in drone-to-drone applications. The main enhanced component of the integrated model is the target surrounding matching mechanism, which consists of tracking quality evaluation and candidate target selection and trajectory prediction modules. In order to improve the reliability of the tracking results, this embodiment introduces additional information outside the tracker output to evaluate the current tracking performance. In addition, in this embodiment, spatiotemporal consistency is used to evaluate the current tracking results, based on response graph analysis and trajectory prediction, and spatiotemporal consistency is used to evaluate the current tracking results;

[0040] On the basis of the above scheme, step S100 may include: obtaining image sequence data in the target drone tracking scene according to a predetermined sampling frequency through an image acquisition device mounted on the tracking drone, wherein the image sequence data is used to show visual information of the target drone and its surrounding environment; inputting the obtained image sequence data into an integrated model, and the OSTrack single-stream basic tracker in the integrated model performs feature extraction operations on a template area and a search area in the image sequence data using a self-attention mechanism; wherein the template area is selected as a specific image area containing the target drone, and the search area is set to a larger range area in the current frame where the target drone may exist; generating a response map based on the extracted feature information, wherein the feature information of the target drone includes but is not limited to color features, texture features, shape features, and relative position features of the target drone and the surrounding environment;

[0041] Step S200: The integrated model evaluates the reliability of the current tracking result based on the principle of spatiotemporal consistency, combined with response graph analysis and trajectory prediction, and constructs a candidate target set according to the evaluation of the reliability of the tracking result, wherein the response graph analysis and trajectory prediction are defined as two evaluation rules respectively.

[0042] Specifically, rule one is response graph analysis, which may include: setting a threshold to identify positions in the response graph with scores higher than the threshold, obtaining candidate target bounding boxes based on a soft non-maximum suppression algorithm, and judging the tracking status according to the number of candidate boxes. At the same time, the OSTrack single-stream basic tracker predicts the target position based on the obtained candidate target bounding boxes; judging the tracking status according to the number of candidate boxes includes: if the number of candidate boxes is one, it is judged that the reliability of the current tracking result is high and it is in a stable tracking state; if the number of candidate boxes is greater than one, it is judged that the reliability of the current tracking result is low, and all targets corresponding to the candidate boxes are included in the candidate target set.

[0043] More specifically, the response graph provides insight into the tracking status. If the response graph shows multiple peaks, it indicates low reliability and the real target may be between these peaks. Inspired by target detection algorithms such as YOLO, this embodiment uses a method combining soft non-maximum suppression (Soft-NMS) and response graphs, which may include:

[0044] set up represents the response graph, is the threshold. The recognition score of the present invention is higher than The positions of , and Soft-NMS is applied to these positions to obtain candidate target bounding boxes. If the number of candidate boxes is 1, it indicates a normal tracking state with a single peak in the response map. If there are multiple candidate boxes, they are all included in the candidate target set.

[0045] Rule 2 is trajectory prediction, which may include: using the Transformer-based trajectory prediction module in the integrated model to obtain the trajectory sequence of the target UAV in the previous multiple frames; the trajectory prediction module predicts the future trajectory of the target UAV in the next frame based on the acquired trajectory sequence of the target UAV in the previous multiple frames; comparing the predicted position of the trajectory prediction module with the target position predicted by the OSTrack single-stream basic tracker, and after calculating the Euclidean distance, comparing the calculated result with a preset distance threshold. If the calculated result is not higher than the preset distance threshold, the predicted position of the trajectory prediction module is determined to be normal, otherwise it is determined to be abnormal, and the predicted result is included in the candidate target set as a candidate target.

[0046] More specifically, a Transformer-based trajectory prediction module is used as a supplementary judgment criterion. The trajectory sequence of the target is obtained in the frame, expressed as , and predict the next Future trajectory of frames: .in, The first point in is used as the predicted position for the current frame. This embodiment then calculates the difference between this predicted position and the tracker's predicted position :

[0047]

[0048] represents the Euclidean norm, which calculates the predicted position and tracker predicted position If ( is a predefined threshold), the prediction of the tracker is considered normal; otherwise, it is considered abnormal. If these two rules are met, the prediction of the current tracker is considered reliable. If either rule fails, the prediction is considered unreliable and the current candidate target set is recorded. If rule two fails, the trajectory prediction result will be added to the candidate target set as one of the candidates.

[0049] Step S300: calculating a consistency score for the trajectory sequence generated by each candidate target in the candidate target set, selecting the candidate target with the highest score, and taking its initial position as the target position;

[0050] Step S300 may include: determining that each candidate target generates a trajectory sequence ;

[0051] The consistency score is calculated using the following formula: :

[0052] ;

[0053] in, is the candidate target selection unit, Indicates Candidates in frame The location of , , represents the total number of candidate targets, is the total length of the time series, indicating the total number of frames from the first frame to the last frame, is the upper limit of the sum, i.e., the total number of frames considered when calculating the consistency score;

[0054] Among them, the candidate location is evaluated and the expected true target position The candidate with the highest score is determined as the most likely target, and the initial position corresponding to the candidate is used as the position of the target.

[0055] Step S400: Reverse the trajectory sequence corresponding to the target position, reversely predict the trajectory, and calculate the mean square error between the predicted position and the actual position of each reverse predicted trajectory in each frame.

[0056] Step S400 may include: arranging the trajectory sequence corresponding to the determined target position in reverse order according to the time sequence, so that the last frame of the original trajectory sequence becomes the starting frame, and the starting frame becomes the last frame; inputting the reversed trajectory sequence into a trajectory prediction module based on Transformer, the trajectory prediction module predicts the possible position of the target drone on each reverse frame by weighted fusion of information at different positions based on the spatiotemporal correlation characteristics between the frames in the trajectory sequence; for each reverse predicted trajectory, obtaining its predicted position on each frame, and at the same time, extracting the actual position information of the target drone on the corresponding frame from the original image sequence data; according to the mean square error formula, calculating the mean square error between the predicted position and the actual position of each reverse predicted trajectory on each frame, the mean square error is used to show the degree of deviation between the reverse predicted trajectory and the actual trajectory, such as Figure 3 As shown, it is a schematic diagram of the actual test of the method of the present invention.

[0057] On the basis of the above scheme, The MSE calculation formula for candidate targets is as follows:

[0058] ·

[0059] in, The current target in the reverse history track The position of the frame, is the corresponding predicted position, is the total length of the time series, indicating the total number of frames from the first frame to the last frame. According to the principle of spatiotemporal consistency, the candidate target with the smallest MSE is selected as the true target. for:

[0060] ;

[0061] Selected candidate targets It is considered to be the candidate target that best matches the spatiotemporal characteristics of the real target.

[0062] Step S500: Select the candidate target with the smallest mean square error as the real target, so as to obtain the final tracking result, so as to improve the accuracy and reliability of drone-to-drone tracking.

[0063] Based on this embodiment, the present invention proposes an overall framework for enhanced Transformer-based UAV tracking using interference elimination and trajectory prediction (ETUT), which is as follows: Figure 2 As shown in the figure, the proposed framework selects OSTrack as the single-stream base tracker and integrates OSTrack with the target surrounding matching mechanism and the Transformer-based trajectory prediction module to ensure robust and accurate tracking in UAV-to-UAV applications. The main enhanced component of the framework is the target surrounding matching mechanism, which consists of tracking quality assessment and candidate target selection and trajectory prediction modules.

[0064] Furthermore, in the process of implementing the solution of the present invention, based on the time-space consistency, for the same target, in order to ensure real-time performance, the present invention uses multi-threaded parallel processing.

[0065] This paper first proposes a Transformer-based robust UAV-to-UAV tracking framework to improve tracking accuracy by eliminating interference and trajectory prediction. The framework utilizes a single-stream Transformer tracker and integrates key components, including a target surrounding matching mechanism and a Transformer-based trajectory prediction module. The Transformer tracker quickly generates proposals, while the enhancement component verifies these proposals to determine the most relevant ones to the target. Secondly, this paper introduces a target surrounding matching mechanism to effectively eliminate interference factors and ensure that the tracker can still focus on the relevant target in the presence of background noise and similar objects.

[0066] Finally, we propose a Transformer-based trajectory prediction module designed to manage the highly dynamic motion between UAVs. This module can accurately predict the target’s position even if it is temporarily out of sight of the tracker, thus reducing the risk of tracking failure.

[0067] The specific embodiments of the present invention are further described below.

[0068] First, we exploit spatiotemporal consistency to correct the appearance model built by the tracking network when it is deemed unreliable. This process can include:

[0069] When the reliability estimation module detects an unreliable track, it generates suspicious candidate targets. According to the spatiotemporal consistency, for the same target, forward tracking is performed along the sequence, and then backward tracking is performed, and ideally it should return to the original target position. To ensure real-time performance, multi-threaded parallel processing can be used to handle this process.

[0070] Each candidate target generates a trajectory sequence , this method calculates the consistency score , to measure whether the trajectory conforms to the expected motion pattern of the target. The candidate with the highest score is selected as the most likely target. The initial position corresponding to the candidate is then used as the position of the target. Formally, the consistency score It can be calculated as:

[0071]

[0072] in is the candidate target selection module, Indicates Candidates in frame The location of , , which evaluates the candidate locations and the expected true target position By utilizing this enhanced framework, the proposed method can achieve more robust and accurate tracking performance in UAV-to-UAV scenarios, mitigate the disadvantage of relying on confidence scores, and improve the reliability of the tracking system.

[0073] The present invention selects OSTrack as the single-stream basic tracker, and the specific process is as follows:

[0074] OSTrack overcomes the limitations of traditional tracking frameworks by unifying feature learning and relation modeling. The integration of templates and search images enables a bidirectional flow of information, dynamically extracting discriminative target features through mutual guidance.

[0075] OSTrack uses Vision Transformer (ViT), which extracts features from both the template and the search region simultaneously by stacking self-attention layers. The self-attention mechanism in ViT promotes the free flow of information between the template and the search region, enhancing the discriminative power of the extracted features. It can be defined as:

[0076] ;

[0077] Where Q, K and V are query matrix, key matrix and value matrix respectively, and dk is the dimension of the key. The design of the present invention allows efficient and parallelizable computation, improving performance and high frame rate.

[0078] Finally, the present invention introduces a target surrounding matching mechanism and a trajectory prediction module, which may include:

[0079] 1) In order to improve the reliability of the tracking results, the present invention introduces additional information beyond the tracker output to evaluate the current tracking performance. The present invention uses spatiotemporal consistency to evaluate the current tracking results according to the following two rules.

[0080] Rule 1: The response graph provides insight into the tracking status. If the response graph shows multiple peaks, it indicates low reliability and the real target may be between these peaks. Inspired by target detection algorithms such as YOLO, this paper adopts a method combining soft non-maximum suppression (Soft-NMS) and response graph for this purpose.

[0081] set up represents the response graph, is the threshold. The recognition score of the present invention is higher than The positions of the candidate boxes are obtained by applying soft NMS to these positions to obtain candidate target bounding boxes. If the number of candidate boxes is 1, it indicates a normal tracking state with a single peak in the response map. If there are multiple candidate boxes, they are all included in the candidate target set.

[0082] Rule 2: In order to handle the situation where the response graph fails due to appearance model degradation, the present invention uses a Transformer-based trajectory prediction module as a supplementary judgment criterion. This module is derived from the previous The trajectory sequence of the target is obtained in the frame, expressed as , and predict the next The future trajectory of the frame, . The first point in is used as the predicted position of the current frame. The present invention then calculates the difference between this predicted position and the tracker's predicted position :

[0083]

[0084] in, represents the Euclidean norm, which calculates the predicted position and tracker predicted position If ,in is a predefined threshold, the tracker’s prediction is considered normal; otherwise, it is considered anomaly.

[0085] If both rules are met, the prediction of the current tracker is considered reliable. If either rule fails, the prediction is considered unreliable and the current candidate target set is recorded. If rule 2 fails, the trajectory prediction result will be added to the candidate target set as one of the candidates.

[0086] By combining response map analysis with trajectory prediction, we robustly evaluate the reliability of tracking results and construct a candidate target set to ensure accurate and reliable tracking in UAV-to-UAV scenarios.

[0087] 2) The present invention introduces a Transformer-based trajectory prediction module. By leveraging the powerful continuous data modeling capabilities of Transformer and performing end-to-end offline training specifically for drone-to-drone tracking scenarios, the present invention achieves accurate trajectory prediction.

[0088] To enable the Transformer to capture the order of the input sequence, we add sinusoidal positional encodings to the input embedding. and dimension , the position encoding is defined as follows:

[0089]

[0090]

[0091] in is the dimension of the input embedding. These positional encodings are added to the input embedding to contain the position information of the sequence.

[0092] The main model uses a stack of Transformer encoder layers, each of which consists of a multi-head self-attention mechanism and a position feedforward neural network. The input of the Transformer encoder layer is the previous The trajectory sequence of frames is expressed as .

[0093] Each Transformer encoder layer performs a multi-head self-attention operation and a position feed-forward neural network. The encoder processes the input trajectory sequence through multiple layers to capture the complex dependencies between elements. Represents the output of the final Transformer encoder layer.

[0094] By integrating a Transformer-based trajectory prediction module, our method can accurately predict the future position of a target and achieve robust tracking even in highly dynamic UAV-to-UAV scenarios.

[0095] 3) When the candidate target set Contains multiple candidates ( ), the present invention needs to filter these candidates to select the candidate that best matches the real target based on spatiotemporal consistency. The method of the present invention is to initialize a separate Transformer tracker to track each candidate target.

[0096] Then, the mean square error (MSE) of the trajectory sequence of each candidate target is calculated, and the candidate target with the smallest MSE is selected as the true target. The MSE calculation formula for candidate targets is as follows:

[0097] ·

[0098] in, The current target in the reverse history track The position of the frame, is the corresponding predicted position. According to the principle of spatiotemporal consistency, the candidate target with the smallest MSE is selected as the true target. for:

[0099]

[0100] Selected candidate targets It is considered to be the candidate target that best matches the spatiotemporal characteristics of the real target.

[0101] The electronic device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0102] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the enhanced UAV-to-UAV tracking method of each embodiment of the present disclosure.

[0103] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0104] like Figure 4 The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 4The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 4 As shown, the electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0106] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of an enhanced drone-to-drone tracking method of an embodiment of the present disclosure are executed.

[0108] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0109] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the enhanced drone-to-drone tracking method of the aforementioned embodiments of the present disclosure are executed.

[0110] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0111] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0112] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0113] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0114] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0115] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0116] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0117] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0118] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An enhanced UAV-to-UAV tracking method, characterized in that: The following steps are involved: Acquire image sequence data in a target UAV tracking scene, input the acquired image sequence data into a preset integrated model, extract feature information of the target UAV, and the integrated model generates a response graph according to the feature information of the target UAV; The integrated model is based on the principle of spatiotemporal consistency, combines response graph analysis and trajectory prediction, evaluates the reliability of the current tracking result, and constructs a candidate target set based on the evaluation of the reliability of the tracking result; Calculating a consistency score for the trajectory sequence generated by each candidate target in the candidate target set, selecting the candidate target with the highest score, and taking its initial position as the target position; Reversing the trajectory sequence corresponding to the target position, reversely predicting the trajectory, and calculating the mean square error between the predicted position and the actual position of each reverse predicted trajectory in each frame; The candidate target with the smallest mean square error is selected as the true target to obtain the final tracking result, so as to improve the accuracy and reliability of UAV-to-UAV tracking.

2. The enhanced UAV-to-UAV tracking method according to claim 1, characterized in that: The integrated model includes: OSTrack single-stream base tracker for processing input image sequence data; A target surrounding matching mechanism unit for further analyzing the image sequence data processed by the OSTrack single-stream basic tracker; A trajectory prediction module is used to predict the trajectory of a target UAV, wherein the trajectory prediction module predicts the trajectory of the UAV based on a Transformer.

3. The enhanced UAV-to-UAV tracking method according to claim 2, characterized in that: The step of acquiring image sequence data in a target UAV tracking scene, inputting the acquired image sequence data into a preset integrated model, extracting feature information of the target UAV, and generating a response graph according to the feature information of the target UAV by the integrated model includes: By using an image acquisition device mounted on the tracking drone, image sequence data in a target drone tracking scene is acquired at a predetermined sampling frequency, wherein the image sequence data is used to show visual information of the target drone and its surrounding environment; The acquired image sequence data is input into the integrated model, and the OSTrack single-stream basic tracker in the integrated model uses the self-attention mechanism to perform feature extraction operations on the template area and the search area in the image sequence data; Generate a response map based on the extracted feature information.

4. The enhanced UAV-to-UAV tracking method according to claim 1, characterized in that: The response plot analysis includes, A threshold is set to identify the locations in the response map with scores higher than the threshold. The candidate target bounding boxes are obtained based on the soft non-maximum suppression algorithm. The tracking status is determined according to the number of candidate boxes. At the same time, the OSTrack single-stream basic tracker predicts the target position based on the obtained candidate target bounding boxes.

5. The enhanced UAV-to-UAV tracking method according to claim 4, characterized in that: The determining of the tracking state according to the number of candidate boxes includes: The number of candidate boxes is one, which means that the reliability of the current tracking result is high and the tracking result is stable. If the number of candidate boxes is greater than one, it is determined that the reliability of the current tracking result is low, and the targets corresponding to all the candidate boxes are included in the candidate target set.

6. The enhanced UAV-to-UAV tracking method according to claim 4, characterized in that: The trajectory prediction includes: Using the Transformer-based trajectory prediction module in the integrated model, the trajectory sequence of the target UAV in the previous multiple frames is obtained; The trajectory prediction module predicts the future trajectory of the target UAV in the next frame according to the acquired trajectory sequence of the target UAV in the previous multiple frames; The predicted position of the trajectory prediction module is compared with the target position predicted by the OSTrack single-stream basic tracker. After calculating the Euclidean distance, the calculation result is compared with the preset distance threshold. If the calculation result is not higher than the preset distance threshold, the predicted position of the trajectory prediction module is considered normal, otherwise it is considered abnormal, and the prediction result is included in the candidate target set as a candidate target.

7. The enhanced UAV-to-UAV tracking method according to claim 1, characterized in that: The step of calculating a consistency score of the trajectory sequence generated for each candidate target in the candidate target set, selecting the candidate target with the highest score, and taking its initial position as the target position includes: Determine a trajectory sequence for each candidate target ; The consistency score is calculated using the following formula: : in, is the candidate target selection unit, Indicates Candidates in frame The location of , , represents the total number of candidate targets, is the total length of the time series, indicating the total number of frames from the first frame to the last frame, is the upper limit of the sum, i.e., the total number of frames considered when calculating the consistency score; Among them, the candidate location is evaluated and the expected true target position spatial and temporal alignment between; The candidate target with the highest score is determined to be the most likely target, and the initial position corresponding to the candidate target is used as the position of the target.

8. The enhanced UAV-to-UAV tracking method according to claim 1, characterized in that: The step of reversing the trajectory sequence corresponding to the target position, reversely predicting the trajectory, and calculating the mean square error between the predicted position and the actual position of each reversely predicted trajectory in each frame includes: The trajectory sequence corresponding to the determined target position is reversely arranged in time order, so that the last frame of the original trajectory sequence becomes the starting frame, and the starting frame becomes the last frame; The reversed trajectory sequence is input into the trajectory prediction module based on Transformer. The trajectory prediction module predicts the possible position of the target UAV in each reverse frame by weighted fusion of information at different positions based on the spatiotemporal correlation features between frames in the trajectory sequence. For each reverse prediction trajectory, obtain its predicted position on each frame, and at the same time, extract the actual position information of the target drone on the corresponding frame from the original image sequence data; According to the mean square error formula, the mean square error between the predicted position and the actual position of each of the reverse prediction trajectories in each frame is calculated, and the mean square error is used to show the degree of deviation between the reverse prediction trajectory and the actual trajectory.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the enhanced drone-to-drone tracking method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the enhanced drone-to-drone tracking method described in any one of claims 1-8.

Images