An unmanned aerial vehicle airborne dynamic target recognition method and system

Abstract

The application discloses a kind of unmanned aerial vehicle airborne dynamic target identification method and system, method includes the dense optical flow map of current frame being obtained by calculating two continuous frames of image through optical flow model;Dense optical flow map and current frame RGB image are spliced according to channel, form multi-modal input feature matrix;Through lightweight backbone network model, multi-modal input feature matrix is extracted, and current frame single frame feature pyramid is obtained;Cross-frame channel attention weighted fusion is carried out to previous frame single frame feature pyramid and current frame single frame feature pyramid, and time sequence feature pyramid is obtained;Based on time sequence feature pyramid, the class and position information of dynamic target are output by YOLO11 target prediction model.The application is through two-stage fusion and three-stage progressive training and recursive feature reuse, under the condition of airborne low algorithm, the robustness and real-time performance of dynamic identification in complex airborne scene are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of dynamic target recognition technology for unmanned aerial vehicles (UAVs), and particularly to an airborne dynamic target recognition method and system for UAVs. Background Technology

[0002] UAV-borne dynamic target recognition refers to the process by which an aircraft uses its built-in camera or other sensors to identify and track moving targets on the ground or in the air in real time. This means recognizing the target in a single frame and continuously tracking its location, speed, and trajectory changes over multiple frames. In applications such as inspection, search and rescue, tracking, and collision avoidance, targets rarely remain stationary and wait to be detected. Only by monitoring their movement in real time can UAVs upgrade their ability to see to track, predict, and react promptly.

[0003] Existing UAV-borne dynamic target recognition technology has the following limitations: 1. Traditional YOLO series algorithms (including YOLO11) mainly rely on single-frame image features for target detection, and do not effectively explore the temporal correlation between consecutive frames. This leads to a sharp drop in the recognition accuracy of dynamic targets (such as fast-moving vehicles and pedestrians) when their motion state changes abruptly (such as acceleration or turning), and a lack of feature continuity in the time dimension.

[0004] 2. Existing methods mostly focus on extracting the appearance features of the target, neglecting the ability of optical flow information to characterize the motion trajectory. In scenarios where the target is occluded, rotated, or the background is moving (such as rapid ground displacement during low-altitude flight), it is easy to miss or falsely detect the target, resulting in poor adaptability to dynamic scenes.

[0005] 3. In UAV-borne scenarios, there are interference factors such as strong shaking, drastic changes in lighting (such as backlighting and shadows), and atmospheric turbulence. Existing algorithms have not been optimized for the characteristic variation patterns of dynamic targets in such environments, resulting in decreased recognition stability and insufficient robustness to meet the requirements of actual tasks.

[0006] 4. Dynamic targets moving at high speeds are prone to image blurring. Traditional algorithms have limited ability to model blur features and cannot effectively distinguish between "motion blur" and "target shape change", resulting in a significant reduction in recognition accuracy in low-altitude, high-speed tracking scenarios.

[0007] 4. The dynamic target samples in the training dataset mostly come from simple motion scenarios (such as uniform linear motion), lacking coverage of complex modes such as non-uniform speed, change of direction, and sudden motion, resulting in insufficient generalization ability of the algorithm in real dynamic scenarios.

[0008] 5. Some improved algorithms enhance robustness by introducing complex motion modeling, but this increases computational complexity, making it unsuitable for the computing power limitations of the UAV's onboard terminal and unable to meet real-time recognition requirements. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides a method and system for identifying dynamic targets on UAVs. By fusing temporal information and optical flow features, the method optimizes the YOLO11 algorithm's ability to model the motion patterns of dynamic targets, thereby enhancing its robustness and real-time performance in complex airborne scenarios and meeting the practical needs of UAVs for tasks such as dynamic target tracking and obstacle avoidance.

[0010] This invention provides a method for identifying airborne dynamic targets on unmanned aerial vehicles (UAVs), the specific technical solution of which is as follows: S1: Calculate the dense optical flow map of the current frame by using the optical flow model on two consecutive frames; S2: Before the backbone network, the dense optical flow map is concatenated with the current frame RGB image by channel to form a multimodal input feature matrix, thus completing the pre-feature fusion; S3: Extract features from the multimodal input feature matrix using a lightweight backbone network model to obtain the single-frame feature pyramid for the current frame; S4: Perform cross-frame channel attention weighted fusion of the single-frame feature pyramid of the previous frame and the single-frame feature pyramid of the current frame to obtain the temporal feature pyramid, thus completing the feature fusion. S5: Based on the temporal feature pyramid, output the category and location information of dynamic targets through the YOLO11 target prediction model.

[0011] Furthermore, the optical flow model uses a lightweight FlowNet2 as its backbone, connected to a lightweight backbone network model MobileNetV3-0.33 as its main backbone, and then connected to a YOLO11 target prediction model. The input channel of the YOLO11 target prediction model is a receiving temporal feature pyramid, and the output is the category confidence of the dynamic target, the bounding box coordinates, and the target trajectory information.

[0012] Furthermore, the optical flow model is trained as follows: The endpoint error EPE-weighted loss is used as the loss function for training, which optimizes the model's accuracy in estimating optical flow for fast-moving targets in UAV scenarios.

[0013] Furthermore, the lightweight backbone network model is trained as follows: The model was pre-trained for standard classification using the ImageNet dataset to obtain general feature weights. The feature weights are transferred to the detection score, the first two convolutional layers are frozen, and starting from the third convolutional layer, the network parameters are updated using the pre-trained weights as initial values.

[0014] Furthermore, the training of the YOLO11 target prediction model includes: Static single-frame training: Only RGB image data is input. The input ports of the optical flow graph and the feature pyramid of the previous frame are filled with all-zero tensors. The model is trained with CIoU loss + cross-entropy loss to optimize the model's ability to recognize static targets. Single-frame optical flow fusion training: The single-frame RGB image and the corresponding optical flow map are concatenated by channel and then input into the model. The input ports of the feature pyramid of the previous frame are filled with zeros. The model is trained by multimodal feature fusion loss to enhance the model’s perception of motion information within a single frame. Temporal multi-frame optical flow fusion training: RGB images and optical flow sequences of several consecutive frames are fed into the network in chronological order. The feature pyramid of the previous frame is used as a reference feature. Based on CIoU loss and cross-entropy loss, temporal consistency loss is introduced for training to optimize the model's ability to model the temporal trajectory of dynamic targets.

[0015] Furthermore, during training in each segment, the learning rate and training rounds are dynamically adjusted based on the accuracy of the validation set.

[0016] Furthermore, before training each model, a hybrid dataset is constructed. The training set input for each model is obtained from this hybrid dataset, which includes real-world scene data and open-source data. The construction process is as follows: Collect and acquire dynamic target video sequences from various real-world scenarios, and manually annotate them to generate labeled data including target category, bounding box, and motion trajectory; We acquire dynamic target samples from open-source datasets and integrate them with labeled data augmentation.

[0017] Furthermore, in steps S3-S4, the high-dimensional features calculated in the previous frame are recursively reused for the continuous frame sequence.

[0018] The present invention also provides an airborne dynamic target recognition system for unmanned aerial vehicles (UAVs), which is applied to the aforementioned airborne dynamic target recognition method for UAVs. The system includes an optical flow generation module, a lightweight backbone network, and a target prediction module. The optical flow generation module is connected to the lightweight backbone network, and the output of the lightweight backbone network is connected to the target prediction module. The optical flow generation module stores an optical flow model with the backbone network removed, which is used to calculate and output a dense optical flow map of two consecutive frames of input images. The lightweight backbone network includes a first feature fusion module and a feature extraction module; The first feature fusion module concatenates the dense optical flow map with the current frame RGB image by channel; the feature extraction module extracts the single-frame feature pyramid of the current frame based on the concatenated image. The target prediction module stores a YOLO11 target prediction model, including a second feature fusion module and a prediction output module; The second feature fusion module is used to perform cross-frame channel attention weighted fusion processing on the single-frame feature pyramid of the current frame and the single-frame feature pyramid of the previous frame, and output a temporal feature pyramid. The prediction output module is used to obtain the category and location of dynamic targets based on the temporal feature pyramid.

[0019] Furthermore, the lightweight backbone network, optical flow model, and YOLO11 target prediction model are deployed using INT8 and INT16 mixed precision quantization.

[0020] The beneficial effects of this invention are as follows: This invention utilizes optical flow-RGB dual-channel pre-fusion and cross-frame attention temporal fusion to inject motion trajectory and appearance features into the target prediction model simultaneously through two-level fusion. This fills the gap in temporal-motion information missing in single-frame dynamic recognition schemes, fundamentally solving the problem of missed / false detections when dynamic targets accelerate, turn sharply, or are occluded. The entire network model uses MobileNetV3-0.33 as its backbone, combined with recursive feature reuse and mixed precision quantization, enabling recognition to be completed in ≤50 ms under limited onboard computing power. At the same time, a phased training strategy is adopted, which improves the model's adaptability to diverse motion modes through progressive training from static to dynamic. Combined with multiple loss functions for targeted optimization of dynamic interference scenarios, both real-time performance and robustness are achieved. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention are clearly and completely described in the following description. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use, or the orientation or positional relationship in which those skilled in the art conventionally understand it during use. This is only for the convenience of describing the present invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0025] Example 1 Embodiment 1 of the present invention discloses an airborne dynamic target recognition method for unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, the specific process is as follows: S1: Calculate the dense optical flow map of the current frame by using the optical flow model on two consecutive frames; S2: Before the backbone network, the dense optical flow map is concatenated with the current frame RGB image by channel to form a multimodal input feature matrix, thus completing the pre-feature fusion; By preserving the original spatial distribution information of pixel-level motion trajectories, the model's sensitivity to pixel displacement in dynamic scenes is improved by more than 20%, especially enhancing its ability to capture the motion trends of small moving targets (such as slow-moving drones at low altitudes) and targets with sudden changes in direction. S3: Extract features from the multimodal input feature matrix using a lightweight backbone network model to obtain the single-frame feature pyramid for the current frame; S4: The single-frame feature pyramid of the previous frame and the single-frame feature pyramid of the current frame are fused together by cross-frame channel attention weighting to obtain the temporal feature pyramid, thus completing the feature fusion. Through cross-frame feature splicing and channel attention weighting, the target morphology features and motion trend features at different times are directly embedded into the feature map. While preserving the inter-frame correlation, the amount of spatiotemporal information input to the model is increased by 40%-60% compared to single-frame processing, effectively enhancing the ability to capture moving targets. As a preferred embodiment, in steps S3-S4, when extracting features from a continuous frame sequence, the high-dimensional features already calculated in the previous frame are recursively called, and incremental calculation is only performed on the newly added motion information of the current frame. There is no need to repeatedly process the overlapping features between frames. Through the dual optimization of lightweight backbone network and recursive reuse, the computational load of temporal fusion is reduced by 50%-60%. While ensuring the continuous modeling of the dynamic target motion trajectory, the model inference efficiency is significantly improved (inference speed is increased by more than 65%, and end-to-end latency is ≤50ms), which is suitable for the limited computing resources of the UAV on-board terminal.

[0026] S5: Based on the temporal feature pyramid, output the category and location information of dynamic targets through the YOLO11 target prediction model.

[0027] In a preferred embodiment, in the above process, the optical flow model uses a lightweight FlowNet2 as the backbone, connected to a lightweight backbone network model MobileNetV3-0.33 (the number of channels in each layer of the MobileNetV3 model is changed to 1 / 3 of that in the standard MobileNetV3) as the main backbone, and then connected to the YOLO11 target prediction model. The original Neck+Head structure of the YOLO11 target prediction model remains unchanged. Its input channel is the receiving temporal feature pyramid, and the output is the category confidence of the dynamic target, the bounding box coordinates, and the target trajectory information.

[0028] In a preferred embodiment, the optical flow model is trained as follows: In this embodiment, open-source optical flow datasets such as MPI-Sintel and KITTI are used for pre-training to construct an optical flow dataset. The model is trained by using endpoint error EPE weighted loss as the loss function to optimize the optical flow estimation accuracy of fast-moving targets in UAV scenarios.

[0029] As a preferred embodiment, the lightweight backbone network model is trained as follows: The MobileNetV3-0.33 model was pre-trained for standard classification using the ImageNet dataset to obtain general feature weights; The feature weights are transferred to the detection scores, and the first two convolutional layers are frozen. That is, starting from the third convolutional layer, all subsequent intermediate layers and detection heads are updated with the pre-trained weights as the initial values.

[0030] In a preferred embodiment, the YOLO11 target prediction model is trained using a phased, progressive training method, as follows: First, static single-frame training is performed: only RGB image data is input, that is, the input ports of the optical flow map and the feature pyramid of the previous frame are filled with all-zero tensors to ensure that the network can neither see the optical flow nor the features of the previous frame, and can only rely on the current RGB for detection. Then, CIoU loss + cross-entropy loss are used for training to optimize the model's ability to recognize static targets. Then, single-frame optical flow fusion training is performed: the single-frame RGB image and the corresponding optical flow map are concatenated by channel and input into the model. That is, the RGB (3 channels) and the optical flow map (2 channels, x and y displacement) at the same time are concatenated into a 5-channel tensor in the channel dimension. The input port of the feature pyramid of the previous frame is filled with zeros to ensure that the network still cannot see the features of the historical frames and can only use the RGB + optical flow of the current frame. The model is trained by multimodal feature fusion loss (CIoU + cross-entropy + optical flow consistency regularization) to enhance the model's perception of motion information within a single frame. Finally, temporal multi-frame optical flow fusion training is performed: RGB images and optical flow sequences of several consecutive frames are fed into the network in chronological order. The input ports of the feature pyramid of the previous frame are no longer filled with zeros, that is, the single-frame feature pyramid of the previous frame is used as the reference feature. In the first frame, it is still initialized with zeros. In subsequent frames, the true output features of the previous frame are used. Based on CIoU loss and cross-entropy loss, temporal consistency loss (such as inter-frame target position smoothing loss) is introduced for training to optimize the model's ability to model the temporal trajectory of dynamic targets.

[0031] Specifically, during training in each segment, the learning rate and training rounds are dynamically adjusted based on the accuracy of the validation set.

[0032] Before training each model, a hybrid dataset is constructed. The training set input for each model is obtained from this hybrid dataset, which includes real-world scene data and open-source data. The construction process is as follows: Dynamic target video sequences are collected in real-world scenarios such as urban roads, mountainous forests, and low-altitude airspace by drones equipped with high-definition cameras. These sequences are then manually annotated to generate labeled data that includes target categories, bounding boxes, and motion trajectories. Expand the data scale by integrating dynamic target samples from open-source datasets such as COCO and VisDrone; In this embodiment, all video sequences can be processed in batches using the trained FlowNet2 model, and dense optical flow maps between consecutive frames can be calculated offline and saved according to the frame index, thereby forming a three-in-one training data unit of original image-optical flow map-annotation information.

[0033] Example 2 Embodiment 2 of the present invention discloses an airborne dynamic target recognition system for unmanned aerial vehicles (UAVs), which is applied to the airborne dynamic target recognition method for UAVs described in Embodiment 1 above. The system includes an optical flow generation module, a lightweight backbone network, and a target prediction module. The optical flow generation module is connected to the lightweight backbone network, and the output of the lightweight backbone network is connected to the target prediction module. The optical flow generation module stores an optical flow model with the backbone network removed. It uses FlowNet2 as the basic framework and replaces its backbone network with the lightweight backbone network MobileNetV3 0.33. This model is used to calculate and output dense optical flow maps of two consecutive frames of input images. The lightweight backbone network includes a first feature fusion module and a feature extraction module; The first feature fusion module concatenates the dense optical flow map with the current frame RGB image by channel; the feature extraction module extracts the single-frame feature pyramid of the current frame based on the concatenated image. The target prediction module stores a YOLO11 target prediction model. Specifically, the YOLO11 target prediction model is as follows: the original Neck+Head structure remains unchanged, but its input channel is improved to receive a temporal feature pyramid, and the output is the category confidence of the dynamic target, the bounding box coordinates, and the target trajectory information. The target prediction module includes a second feature fusion module and a prediction output module. The second feature fusion module is used to perform cross-frame channel attention weighted fusion processing on the single-frame feature pyramid of the current frame and the single-frame feature pyramid of the previous frame, and output a temporal feature pyramid. The prediction output module is used to obtain the category and location of dynamic targets based on the temporal feature pyramid.

[0034] In a preferred embodiment, the lightweight backbone network, optical flow model, and YOLO11 target prediction model are deployed using a hybrid precision quantization of INT8 and INT16. That is, for the same set of weight parameters, precision-sensitive convolutional layers are stored using 16-bit storage, while other precision-insensitive convolutional / fully connected layers are stored using 8-bit storage. This saves half the storage compared to pure INT16 and has higher precision than pure INT8, significantly improving model processing time while preserving the precision of key layers. In this embodiment, deployment also includes layer fusion optimization using deep learning inference optimization tools. Layer fusion removes redundant layers and merges linear layers to accelerate inference. After the three models are deployed, they are coordinated through a lightweight interface to ensure that the end-to-end inference latency is controlled within 50ms.

[0035] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for identifying dynamic targets onboard a UAV, characterized in that, include: S1: Calculate the dense optical flow map of the current frame by using the optical flow model on two consecutive frames; S2: The dense optical flow map is concatenated with the current frame RGB image by channel to form a multimodal input feature matrix; S3: Extract features from the multimodal input feature matrix using a lightweight backbone network model to obtain the single-frame feature pyramid for the current frame; S4: Perform cross-frame channel attention weighted fusion of the single-frame feature pyramid of the previous frame and the single-frame feature pyramid of the current frame to obtain the temporal feature pyramid; S5: Based on the temporal feature pyramid, output the category and location information of dynamic targets through the YOLO11 target prediction model.

2. The UAV-borne dynamic target recognition method according to claim 1, characterized in that, The optical flow model uses a lightweight FlowNet2 as its backbone, connected to a lightweight backbone network model as its main backbone, and then connected to a YOLO11 target prediction model. The input channel of the YOLO11 target prediction model is a receiving temporal feature pyramid, and the output is the category confidence of the dynamic target, the bounding box coordinates, and the target trajectory information.

3. The UAV-borne dynamic target recognition method according to claim 2, characterized in that, The optical flow model is trained as follows: The endpoint error EPE-weighted loss is used as the loss function for training.

4. The UAV-borne dynamic target identification method according to claim 2, characterized in that, The lightweight backbone network model is trained as follows: The model was pre-trained for standard classification using the ImageNet open-source dataset to obtain general feature weights. The feature weights are transferred to the detection score, the first two convolutional layers are frozen, and starting from the third convolutional layer, the network parameters are updated using the pre-trained weights as initial values.

5. The UAV-borne dynamic target recognition method according to claim 2, characterized in that, The training of the YOLO11 target prediction model includes: Static single-frame training: Only RGB image data is input for training, using object detection loss; Single-frame optical flow fusion training: A single-frame RGB image and its corresponding optical flow map are concatenated by channel and input into the model, and training is performed using multimodal feature fusion loss; Temporal multi-frame optical flow fusion training: RGB images and optical flow sequences of several consecutive frames are fed into the network in chronological order. Based on CIoU loss and cross-entropy loss, temporal consistency loss is introduced for training to optimize the model's ability to model the temporal trajectory of dynamic targets.

6. The UAV-borne dynamic target recognition method according to any one of claims 3-5, characterized in that, Before training each model, a hybrid dataset is constructed. This dataset is used to obtain the training set input for each model during training. The hybrid dataset includes real-world scene data and open-source data. The construction process is as follows: Collect and acquire dynamic target video sequences from various real-world scenarios, and manually annotate them to generate labeled data including target category, bounding box, and motion trajectory; We acquire dynamic target samples from open-source datasets and integrate them with labeled data augmentation.

7. The UAV-borne dynamic target identification method according to claim 5, characterized in that, During training in each segment, the learning rate and training rounds are dynamically adjusted based on the accuracy of the validation set.

8. The UAV-borne dynamic target identification method according to claim 1, characterized in that, In steps S3-S4, the high-dimensional features calculated in the previous frame are recursively reused for consecutive frame sequences.

9. A UAV-borne dynamic target recognition system, characterized in that, The UAV-borne dynamic target recognition method according to any one of claims 1-8 includes an optical flow generation module, a lightweight backbone network, and a target prediction module. The optical flow generation module is connected to the lightweight backbone network, and the output of the lightweight backbone network is connected to the target prediction module. The optical flow generation module stores an optical flow model with the backbone network removed, which is used to calculate and output a dense optical flow map of two consecutive frames of input images. The lightweight backbone network includes a first feature fusion module and a feature extraction module; The first feature fusion module concatenates the dense optical flow map with the current frame RGB image by channel; the feature extraction module extracts the single-frame feature pyramid of the current frame based on the concatenated image. The target prediction module stores a YOLO11 target prediction model, including a second feature fusion module and a prediction output module; The second feature fusion module is used to perform cross-frame channel attention weighted fusion processing on the single-frame feature pyramid of the current frame and the single-frame feature pyramid of the previous frame, and output a temporal feature pyramid. The prediction output module is used to obtain the category and location of dynamic targets based on the temporal feature pyramid.

10. The UAV-borne dynamic target recognition system according to claim 9, characterized in that, The lightweight backbone network, optical flow model, and YOLO11 target prediction model are deployed using INT8 and INT16 mixed precision quantization, and layer fusion optimization processing is performed to remove redundant layers and fuse linear layers.

Images