Large and small model collaborative unmanned aerial vehicle target tracking method

Abstract

The invention provides a large and small model collaborative unmanned aerial vehicle target tracking method, which is realized through three stages: firstly, processing multi-modal data such as images, videos and radar signals by using a multi-modal perception large model to carry out preliminary target identification so as to obtain target position information; secondly, target position information generated by the large model is transmitted to a lightweight KCF tracking algorithm for efficient tracking; and finally, when the target is lost, reactivating the large model through a dynamic feedback mechanism to perform a new round of recognition, and optimizing the overall tracking performance by combining online learning and parameter adjustment. The limitation of a traditional unmanned aerial vehicle target tracking method in the aspects of data intensive training, generalization performance, computing resource consumption and environmental adaptability is solved, and an efficient and high-precision unmanned aerial vehicle target identification and tracking method is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence and unmanned aerial vehicle (UAV), and relates to a UAV target tracking method with large and small models coordinated. Background Art

[0002] With the widespread application of drone technology, especially in the fields of low-altitude monitoring, search and rescue, agricultural plant protection, etc., high-precision target tracking has become one of the key technologies. However, existing drone target tracking methods face many challenges, especially when the environment is dynamic and changeable and the types of targets are complex, the limitations of traditional methods become more and more obvious:

[0003] Data-intensive training requirements: Current mainstream target detection algorithms, such as the YOLO series, rely on large amounts of precisely labeled data sets for training, which not only increases the cost of data preparation, but also limits the generalization ability of the model, making it difficult to cope with diverse target categories.

[0004] Limited generalization performance: Despite their excellent performance on specific datasets, these models’ performance drops dramatically when encountering unseen targets or complex backgrounds, especially when drones are flying at low altitudes, where factors such as ground objects, lighting changes, and occlusions can seriously affect tracking accuracy.

[0005] High consumption of computing resources: In order to achieve high accuracy, many target tracking algorithms require a large amount of computing resources, which is a heavy burden for resource-constrained UAV platforms and limits real-time performance and energy efficiency.

[0006] Poor environmental adaptability: When operating in a low-altitude environment, drones must face changeable weather conditions, complex terrain and rapidly changing lighting conditions, which requires the target tracking algorithm to have stronger environmental adaptability and robustness.

[0007] At present, although a variety of deep learning-based target tracking methods have been proposed, such as region-based candidate methods (such as Faster R-CNN), single-stage detection methods (such as YOLO, SSD) and end-to-end tracking frameworks (such as SiamRPN), they still have obvious shortcomings in solving the above problems. Especially in the field of low-altitude drone applications, the above challenges are more severe, and a more efficient, flexible and adaptable target tracking solution is urgently needed. Summary of the invention

[0008] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, propose a large and small model collaborative UAV target tracking method, and solve the limitations of traditional UAV target tracking methods in data-intensive training, generalization performance, computing resource consumption and environmental adaptability.

[0009] The technical solution of the present invention is: a large and small model coordinated UAV target tracking method, comprising the following steps:

[0010] Using a multimodal perception model based on a deep learning algorithm to process multiple sensor output information from the drone to achieve target recognition and generate target location information. The sensor output information includes images, videos, and radar signals.

[0011] The target position information generated by the multimodal perception large model is transmitted to the KCF tracking algorithm for continuous high-frame-rate tracking of the target position. During the tracking process, when the target is detected to be lost, the feedback mechanism is automatically triggered to reactivate the multimodal perception large model to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm to resume the tracking process.

[0012] Furthermore, the multimodal perception large model is based on the Transformer architecture.

[0013] Furthermore, the multimodal perception model processes the output information from multiple sensors of the drone to achieve target recognition as follows:

[0014] The data collected by various sensors of the UAV are fused and processed through the multimodal perception big model. First, the information of different sensors is preprocessed and synchronized to generate a unified data format. Then, the pre-trained multimodal perception big model is used to extract deep features including color, texture, shape, and thermal imaging from the fused data, and a multidimensional feature vector is generated to represent the multidimensional information of the target. Then, the classification network of the multimodal perception big model realizes target recognition and classification according to the feature vector. Finally, the identified target information is transmitted to the KCF tracking algorithm, and real-time tracking is realized using the features extracted by the multimodal perception big model.

[0015] Furthermore, the fused data includes:

[0016] For the input image information, the multimodal perception model extracts the spatial features of the image through a convolutional neural network to obtain the boundary and shape information of the target;

[0017] For the input video information, the multimodal perception model uses the temporal Transformer layer to analyze the temporal changes between video frames to obtain the target's motion trajectory and speed information;

[0018] For the input radar signal, the multimodal perception model extracts the target's distance and speed information through the signal processing module to obtain depth and azimuth data.

[0019] Furthermore, the feedback mechanism is:

[0020] When it is detected that the target is lost, the KCF tracking algorithm outputs tracking failure information and feeds it back to the multimodal perception large model. When the multimodal perception large model receives the tracking failure information, it automatically obtains the sensor output information at the current moment, re-identifies the target, and updates the target's location information.

[0021] Furthermore, when the multimodal perception large model is reactivated, the following process is also performed: dynamically adjusting the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm.

[0022] Furthermore, the method for dynamically adjusting the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm is as follows:

[0023] Dynamically adjust the weights of the multimodal perception model based on the current lighting conditions, occlusion level, and target motion speed, so that the multimodal perception model can more effectively focus on key features in a specific modality, including key features in vision, infrared, or lidar, to improve detection accuracy;

[0024] According to the changes in the target's motion state and the relative distance between the UAV and the target, the KCF filter bandwidth is adjusted in real time, so that the tracking algorithm can maintain stable tracking of the target in complex environments, thereby achieving coordinated optimization of the multimodal perception large model and the KCF tracking algorithm to adapt to changing UAV operation scenarios.

[0025] A large-small model collaborative UAV target tracking system, including a multi-modal perception large model module and a KCF tracking algorithm module;

[0026] The multimodal perception large model module is used to process the output information of various sensors from the drone to achieve target recognition and generate the location information of the target. The sensor output information includes images, videos and radar signals;

[0027] The KCF tracking algorithm module uses the target position information generated by the multimodal perception large model module as input to perform continuous high-frame rate tracking of the target position. During the tracking process, when it is detected that the target is lost, the feedback mechanism is automatically triggered to send tracking failure information to the multimodal perception large model module, and the multimodal perception large model module is reactivated to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm module to resume the target tracking process.

[0028] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large-small model coordinated unmanned aerial vehicle target tracking method are implemented.

[0029] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for tracking a target of an unmanned aerial vehicle with coordinated large and small models are implemented.

[0030] The beneficial effects of the present invention compared with the prior art are:

[0031] (1) The present invention combines a large multimodal perception model with a small KCF classic tracking model to significantly improve the accuracy of recognition and tracking while ensuring recognition and tracking performance;

[0032] (2) The present invention utilizes a large multimodal perception model to improve recognition accuracy, and reduces computing power requirements by using a combination of large and small models, thereby improving the real-time performance and energy efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a large-small model coordinated UAV target tracking method of the present invention;

[0034] Figure 2 This is a framework diagram of a large and small model collaborative UAV target tracking method of Example 1. DETAILED DESCRIPTION

[0035] like Figure 1 The present invention proposes a large-small model coordinated UAV target tracking method, comprising the following steps:

[0036] Step 1: Use a multimodal perception model based on a deep learning algorithm to process multiple sensor output information from the drone to achieve target recognition and generate target location information. The sensor output information includes images, videos, and radar signals.

[0037] Step 2: The target position information generated by the multimodal perception large model is transmitted to the KCF (Kernelized Correlation Filters) tracking algorithm for continuous high-frame-rate tracking of the target position. During the tracking process, when the target is detected to be lost, the feedback mechanism is automatically triggered to reactivate the multimodal perception large model to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm to resume the tracking process.

[0038] In step 1, the multimodal perception model processes the output information from multiple sensors of the drone to achieve target recognition:

[0039] The data collected by various sensors of the UAV (such as RGB cameras, infrared sensors, lidar, etc.) are fused and processed through the multimodal perception large model. First, the information of different sensors is preprocessed and synchronized to generate a unified data format; then, the pre-trained multimodal perception large model is used to extract deep features including color, texture, shape, and thermal imaging from the fused data, and a multidimensional feature vector is generated to represent the multidimensional information of the target; then, the classification network of the multimodal perception large model realizes target recognition and classification according to the feature vector, such as distinguishing between categories such as vehicles, pedestrians, and animals; finally, the identified target information is transmitted to the KCF tracking algorithm, and real-time tracking is achieved using the features extracted by the multimodal perception large model, thus forming an efficient and stable target tracking method with the collaboration of large and small models.

[0040] In step 2, when the multimodal perception large model is reactivated, the following process is also performed: dynamically adjust the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm. The details are as follows:

[0041] First, according to factors such as the current lighting conditions, occlusion level, and target movement speed, the weights of the multimodal perception large model are dynamically adjusted so that the multimodal perception large model can more effectively focus on the key features in a specific modality (such as vision, infrared, or lidar) to improve detection accuracy. Then, according to the changes in the target's motion state (such as acceleration, speed, and direction changes) and the relative distance between the UAV and the target, the KCF filter parameters are adjusted in real time so that the tracking algorithm can maintain stable tracking of the target in a complex environment, thereby achieving coordinated optimization of large and small models and adapting to changing UAV operation scenarios.

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] Example 1

[0044] like Figure 2 As shown, a large-small model coordinated UAV target tracking method of this embodiment includes:

[0045] Step 1: Use a multimodal perception large model based on a deep learning algorithm to process the output information of various sensors from the drone to achieve target recognition and generate the location information of the target. The sensor output information includes images, videos, and radar signals.

[0046] In this embodiment, the drone is equipped with a high-definition camera, infrared sensor and radar equipment to collect images, videos and radar signals. The multimodal perception model based on the deep learning algorithm processes these data and comprehensively analyzes the output information of multiple sensors to achieve preliminary identification of the missing person (target), such as through image recognition of appearance features, infrared sensor detection of human body heat signals, and radar signals to determine location information. The details are as follows:

[0047] The multimodal perception model is based on the Transformer architecture, and the input information includes images, videos and radar signals;

[0048] For the input image information, the model extracts the spatial features of the image through a convolutional neural network (CNN) to obtain the boundary and shape information of the target;

[0049] For the input video information, the model uses the temporal Transformer layer to analyze the temporal changes between video frames to obtain the target's motion trajectory and speed information;

[0050] For the input radar signal, the model extracts the target's distance and speed information through the signal processing module to obtain the depth and azimuth data.

[0051] The multimodal perception model fuses the obtained image, video and radar information, combines the advantages of each modal information, and determines the precise location information of the target.

[0052] Step 2: Transmit the target position information generated by the multimodal perception model to the KCF tracking algorithm to perform continuous high-frame rate tracking of the target position and update the target position in real time.

[0053] The target position generated by the large multimodal perception model is passed to the lightweight KCF tracking algorithm, which is used to maintain efficient real-time tracking performance. It combines the target information provided by the large model to reduce the probability of mistracking and target loss, and updates the target position in real time, maintaining accurate tracking even in complex conditions such as building shadows.

[0054] Step 3: During the tracking process, when the target is detected to be lost, the feedback mechanism is automatically triggered to reactivate the multimodal perception model to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm to resume the tracking process.

[0055] When KCF is unable to continue tracking effectively due to reasons such as tree occlusion, the feedback mechanism is automatically triggered to activate the multimodal perception model to conduct a comprehensive scan and identification of the current environment, and use the newly acquired sensor output information as input to re-identify the target and restore lock on the target.

[0056] The feedback mechanism is as follows: when the target is detected to be lost, the KCF tracking algorithm outputs tracking failure information and feeds it back to the multimodal perception large model. When the multimodal perception large model receives the tracking failure information, it automatically obtains the sensor output information at the current moment, re-identifies the target, and updates the target's location information.

[0057] In this embodiment, when the multimodal perception large model is reactivated, the following process is also performed: dynamically adjusting the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm to adapt to the tracking requirements in different scenarios and further optimize the tracking performance.

[0058] The specific method of dynamically adjusting the weight of the multimodal perception large model and the filter parameters of the KCF tracking algorithm is as follows:

[0059] According to the changes in the drone's surrounding environment, such as light intensity, the frequency of obstructions, the target's movement speed, and other factors, the weights of different modes of the multimodal perception model are adjusted in real time. For example, when the light is low or there is an obstruction, the weight of radar signals or infrared data is increased to enhance the stability of target detection; when the light is sufficient and the vision is clear, the weight of image and video data is increased to enhance the model's ability to recognize detailed features.

[0060] In the KCF tracking algorithm, the filter parameters are dynamically adjusted according to the target's motion state (such as acceleration, speed, and direction change) and the relative distance between the drone and the target. For example, when the target moves quickly or changes direction frequently, the filter bandwidth is increased to enhance the tracking response speed; when the target is relatively still or moves slowly, the bandwidth is reduced to improve the filtering accuracy and avoid tracking jitter.

[0061] The feedback mechanism compares and analyzes the tracking results with the actual target position, and dynamically adjusts the parameters of the multimodal perception model and the KCF tracking algorithm. When the model has tracking deviations, it automatically increases its reliance on other modal information or adjusts the filter parameters to correct the tracking trajectory, thereby ensuring the accuracy and stability of target tracking.

[0062] The whole process above constitutes a closed-loop tracking system. Through the synergy of the large model and the lightweight model, it shows good adaptability and stability in the face of changing target behaviors or external environmental conditions, and can dynamically adjust the parameters of the large model and KCF algorithm according to the tracking effect to optimize performance. It solves the limitations of traditional UAV target tracking methods in data-intensive training, generalization performance, computing resource consumption, and environmental adaptability.

[0063] The present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.

[0064] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0069] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0070] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A large-scale model-coordinated UAV target tracking method, characterized in that: The following steps are involved: Using a multimodal perception model based on a deep learning algorithm to process multiple sensor output information from the drone to achieve target recognition and generate target location information. The sensor output information includes images, videos, and radar signals. The target position information generated by the multimodal perception large model is transmitted to the KCF tracking algorithm for continuous high-frame-rate tracking of the target position. During the tracking process, when the target is detected to be lost, the feedback mechanism is automatically triggered to reactivate the multimodal perception large model to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm to resume the tracking process.

2. The large-small model coordinated UAV target tracking method according to claim 1 is characterized in that: The multimodal perception large model is based on the Transformer architecture.

3. The large-small model coordinated UAV target tracking method according to claim 2 is characterized in that: The multimodal perception model processes the output information from multiple sensors of the drone to achieve target recognition in the following ways: The data collected by various sensors of the UAV are fused and processed through the multimodal perception big model. First, the information of different sensors is preprocessed and synchronized to generate a unified data format. Then, the pre-trained multimodal perception big model is used to extract deep features including color, texture, shape, and thermal imaging from the fused data, and a multidimensional feature vector is generated to represent the multidimensional information of the target. Then, the classification network of the multimodal perception big model realizes target recognition and classification according to the feature vector. Finally, the identified target information is transmitted to the KCF tracking algorithm, and real-time tracking is realized using the features extracted by the multimodal perception big model.

4. The large-small model coordinated UAV target tracking method according to claim 3 is characterized in that: The fused data includes: For the input image information, the multimodal perception model extracts the spatial features of the image through a convolutional neural network to obtain the boundary and shape information of the target; For the input video information, the multimodal perception model uses the temporal Transformer layer to analyze the temporal changes between video frames to obtain the target's motion trajectory and speed information; For the input radar signal, the multimodal perception model extracts the target's distance and speed information through the signal processing module to obtain depth and azimuth data.

5. The large-small model coordinated UAV target tracking method according to claim 1, characterized in that: The feedback mechanism is: When it is detected that the target is lost, the KCF tracking algorithm outputs tracking failure information and feeds it back to the multimodal perception large model. When the multimodal perception large model receives the tracking failure information, it automatically obtains the sensor output information at the current moment, re-identifies the target, and updates the target's location information.

6. The large-small model coordinated UAV target tracking method according to claim 1, characterized in that: When the multimodal perception large model is reactivated, the following process is also performed: the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm are dynamically adjusted.

7. The large-small model coordinated UAV target tracking method according to claim 6, characterized in that: The method for dynamically adjusting the weights of the multimodal perception large model and the filter parameters of the KCF tracking algorithm is as follows: Dynamically adjust the weights of the multimodal perception model based on the current lighting conditions, occlusion level, and target motion speed, so that the multimodal perception model can more effectively focus on key features in a specific modality, including key features in vision, infrared, or lidar, to improve detection accuracy; According to the changes in the target's motion state and the relative distance between the UAV and the target, the KCF filter bandwidth is adjusted in real time, so that the tracking algorithm can maintain stable tracking of the target in complex environments, thereby achieving coordinated optimization of the multimodal perception large model and the KCF tracking algorithm to adapt to changing UAV operation scenarios.

8. A large-small model coordinated UAV target tracking system based on the method according to any one of claims 1 to 7, characterized in that: Including multimodal perception large model module and KCF tracking algorithm module; The multimodal perception large model module is used to process the output information of various sensors from the drone to achieve target recognition and generate the location information of the target. The sensor output information includes images, videos and radar signals; The KCF tracking algorithm module uses the target position information generated by the multimodal perception large model module as input to perform continuous high-frame rate tracking of the target position. During the tracking process, when it is detected that the target is lost, the feedback mechanism is automatically triggered to send tracking failure information to the multimodal perception large model module, and the multimodal perception large model module is reactivated to process the current sensor output information and identify the target, obtain the updated target position information and pass it to the KCF tracking algorithm module to resume the target tracking process.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images