A real-time target detection streaming perception method based on optical flow guidance

Abstract

The application relates to a real-time target detection streaming perception method based on optical flow guidance, which comprises the following steps: extracting multi-scale features of two continuous frames of images; calculating an optical flow field of the two continuous frames of images; selecting optical flow information of a neighborhood around a target in the optical flow field according to a target detection result of a current frame; performing position prediction by using the multi-scale features and the selected optical flow information; and optimizing the predicted position to obtain a final target detection result. The application improves the detection precision and real-time performance of fast-moving objects in UAV videos.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a real-time target detection streaming sensing method based on optical flow guidance. Background Technology

[0002] Real-time object detection in UAV video is an important and challenging task. Traditional object detection methods often suffer from latency issues when dealing with fast-moving objects and dynamic scenes, making it difficult to meet real-time requirements. To address this issue, researchers have proposed the concept of streaming sensing, which aims to predict the future position of objects by analyzing historical data, thereby achieving real-time updates of object states.

[0003] Existing streaming sensing methods, such as Streamer, StreamYOLO, LongShortNet, and DAMO-StreamNet, have made some progress in reducing latency and improving detection accuracy. However, these methods still face challenges when processing fast-moving objects in UAV videos, particularly in capturing fine-grained motion information between consecutive frames. For example:

[0004] StreamYOLO directly inputs two consecutive frames into the network, but lacks a dedicated mechanism to extract and process motion information.

[0005] Although DAMO-StreamNet uses a dual-branch structure to extract semantic and temporal features, it still falls short in capturing key motion cues, especially when dealing with complex trajectories.

[0006] These methods struggle to accurately represent continuous motion between frames, leading to inaccurate detection or object loss in UAV scenarios. In UAV surveillance applications, the wide field of view and rapidly changing flight paths of cameras can cause subtle but significant changes in the appearance and position of objects, further exacerbating this challenge. Summary of the Invention

[0007] The purpose of this invention is to propose a real-time target detection streaming perception method based on optical flow guidance to solve the problems of existing technologies, such as difficulty in accurately capturing fine-grained motion information between consecutive frames, poor performance when handling fast-moving objects and complex trajectories, and difficulty in adapting to subtle but significant changes in the appearance and position of objects in UAV scenes. By integrating optical flow information, the real-time target detection capability is enhanced, and the prediction accuracy of the position of objects in UAV videos is improved.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] A real-time target detection streaming sensing method based on optical flow guidance, comprising:

[0010] Extracting multi-scale features from two consecutive frames of images;

[0011] Calculate the optical flow field of two consecutive frames;

[0012] Based on the target detection results of the current frame, select the optical flow information of the neighborhood surrounding the target in the optical flow field;

[0013] Location prediction is performed using the aforementioned multi-scale features and selected optical flow information;

[0014] The predicted location is optimized to obtain the final target detection result.

[0015] Optionally, extracting multi-scale features from two consecutive frames includes:

[0016] Basic features of two consecutive frames of images were extracted using the YOLOX-L backbone network.

[0017] Based on the aforementioned fundamental features, FPN is used to fuse features from different levels to generate multi-scale features.

[0018] Optionally, calculating the optical flow field of two consecutive frames includes:

[0019] Calculate the optical flow field of two consecutive frames using FastFlowNet;

[0020] The optical flow field is normalized.

[0021] Optionally, location prediction using the multi-scale features and selected optical flow information includes:

[0022] A dynamic motion predictor is built based on a convolutional neural network;

[0023] The multi-scale features and selected optical flow information are input into the dynamic motion predictor to obtain the predicted displacement vector of the target.

[0024] Update the target position using the predicted displacement vector;

[0025] The updated location is combined with the target localization information from the YOLOX-L backbone network to obtain the final predicted bounding box coordinates.

[0026] Optionally, optimizing the predicted location includes:

[0027] For the predicted displacement vector, calculate the corresponding weights;

[0028] Based on the weights, the weighted displacement prediction error is obtained;

[0029] The predicted position is optimized by using the displacement prediction error and the preset target detection loss function.

[0030] Optionally, the corresponding weight is:

[0031]

[0032] Among them, w j As weight, Let x be the square of the predicted displacement in the x-direction, where x is the horizontal coordinate, j is the target index, and ∈ represents the stability term.

[0033] Optionally, the weighted displacement prediction error is:

[0034]

[0035] Among them, G j It is the actual displacement vector, D j It is the predicted displacement vector. M represents the weighted displacement prediction error, and M represents the number of targets.

[0036] Optionally, the method for updating the target position using the predicted displacement vector is as follows:

[0037]

[0038] in, For the updated target location, D represents the target position in the current frame. j It is the predicted displacement vector.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention utilizes the multi-scale features and selected optical flow information for position prediction; optimizes the predicted position to obtain the final target detection result; achieves reliable optical flow information extraction and utilization, enhancing feature consistency between video frames; improves the detection accuracy and real-time performance of fast-moving objects in UAV videos; experiments on multiple challenging UAV video datasets demonstrate that StreamFlow outperforms existing methods in streaming perception tasks. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1This is a schematic diagram of a real-time target detection streaming sensing method based on optical flow guidance according to an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The main components in this embodiment include:

[0046] Multi-scale feature extraction network: It consists of a feature pyramid network (FPN) and a YOLOX-L backbone network.

[0047] Flow-guided dynamic prediction (FGDP) module: Predicts object trajectories using temporal optical flow data.

[0048] Optical Flow Guided Optimization (OFGO) module: Optimizes network parameters by taking into account localization loss and optical flow estimation reliability.

[0049] The Adaptive Stream Weights (AFW) module adjusts the influence of stream samples during training based on their reliability.

[0050] Object detection branch: Processes the fused features and outputs the object detection results.

[0051] Streaming sensing branch: Refine the prediction results using the FGDP module.

[0052] Optical flow estimation module: calculates the optical flow field between consecutive frames.

[0053] Dynamic Motion Predictor (DMP): Generates offset predictions using selected features to refine the initial target detection results.

[0054] The experimental environment for this embodiment is as follows:

[0055] Hardware: A standard computer vision platform equipped with four NVIDIA RTX 3090 GPUs was used.

[0056] Software environment: Implemented based on the deep learning framework PyTorch.

[0057] This embodiment selects four widely used UAV video object detection datasets for experiments:

[0058] UAVDT dataset: Contains approximately 80,000 frames of images from over 10 hours of urban UAV footage, with vehicle annotations and attributes such as weather conditions and flight altitude.

[0059] The AUAIR dataset contains over 32,000 labeled frames from more than 2 hours of RGB video, providing annotations for 8 traffic-related object categories and metadata such as GPS coordinates and IMU sensor data.

[0060] UAV123 dataset: Contains 123 video sequences, over 110,000 frames, designed for aerial tracking challenges, with fully annotated upright bounding boxes.

[0061] The VisDrone dataset contains 288 video clips, totaling 261,908 frames and 10,209 still images from 14 Chinese cities, including over 2.6 million bounding boxes.

[0062] To comprehensively evaluate prediction accuracy and real-time performance, the following evaluation metrics were used in this embodiment:

[0063] Streaming average accuracy (sAP): assesses the accuracy and timeliness of forecasts. Includes different scales (sAP). l ,sAP m ,sAP s ) and IoU threshold (sAP) 50 ,sAP 75 Sub-indices of ).

[0064] Offline average accuracy (off AP): Evaluates target detection performance without considering streaming constraints.

[0065] Velocity-Aware Flow Average Precision (VsAP): Evaluates performance at different object velocities.

[0066] The experimental configuration for this embodiment is as follows:

[0067] Model size: StreamFlow model based on YOLOX-L.

[0068] Learning rate: The initial learning rate is 0.001, and it decreases by 0.1 per epoch.

[0069] Batch size: Each batch contains 16 samples.

[0070] Optimizer: Adam optimizer is used.

[0071] Training epochs: The model is trained for 8 epochs.

[0072] Other parameters: consistent with StreamYOLO.

[0073] The real-time target detection streaming sensing method based on optical flow guidance proposed in this embodiment includes:

[0074] Extracting multi-scale features from two consecutive frames of images;

[0075] Calculate the optical flow field of two consecutive frames;

[0076] Based on the target detection results of the current frame, select the optical flow information of the neighborhood surrounding the target in the optical flow field;

[0077] Location prediction is performed using the aforementioned multi-scale features and selected optical flow information;

[0078] The predicted location is optimized to obtain the final target detection result.

[0079] Furthermore, extracting multi-scale features from two consecutive frames of images includes:

[0080] Basic features of two consecutive frames of images were extracted using the YOLOX-L backbone network.

[0081] Based on the aforementioned fundamental features, FPN is used to fuse features from different levels to generate multi-scale features.

[0082] Specifically, in this embodiment, multi-scale feature extraction includes:

[0083] Input two consecutive frames of images I t-1 and I t .

[0084] Basic features were extracted using the YOLOX-L backbone network.

[0085] FPN is used to fuse features from different levels to generate multi-scale feature maps. Where L represents the number of layers in the feature pyramid.

[0086] Furthermore, calculating the optical flow field of two consecutive frames includes:

[0087] Calculate the optical flow field of two consecutive frames using FastFlowNet;

[0088] The optical flow field is normalized.

[0089] Specifically, in this embodiment, optical flow estimation includes:

[0090] Input two consecutive frames of images I t-1 and I t .

[0091] The optical flow field O = (O) was calculated using FastFlowNet. x O y ).

[0092] The optical flow field is normalized.

[0093] Furthermore, location prediction using the multi-scale features and selected optical flow information includes:

[0094] A dynamic motion predictor is built based on a convolutional neural network;

[0095] The multi-scale features and selected optical flow information are input into the dynamic motion predictor to obtain the predicted displacement vector of the target.

[0096] Update the target position using the predicted displacement vector;

[0097] The updated location is combined with the target localization information from the YOLOX-L backbone network to obtain the final predicted bounding box coordinates.

[0098] Specifically, in this embodiment, the Flow-Guided Dynamic Prediction (FGDP) module implements:

[0099] Optical flow region selection:

[0100] Based on the target detection results of the current frame Where M is the number of targets detected.

[0101] For each detected target j, optical flow information of its surrounding 3×3 neighborhood is selected in the optical flow field O.

[0102] Dynamic Motion Predictor (DMP):

[0103] Design a convolutional neural network C to predict displacement vectors.

[0104] The input to a DMP includes a multi-scale feature map F and selected optical flow information O. j .

[0105] For each target j, the predicted displacement vector is: D j =C(F,O) j ;Θ D ), where Θ D These are learnable parameters of DMP.

[0106] Location refinement:

[0107] Update the target position using the predicted displacement vector:

[0108] The updated location is combined with the target localization information from the YOLOX backbone network to obtain the final predicted bounding box coordinates.

[0109] Further optimization of the predicted location includes:

[0110] For the predicted displacement vector, calculate the corresponding weights;

[0111] Based on the weights, the weighted displacement prediction error is obtained;

[0112] The predicted position is optimized by using the displacement prediction error and the preset target detection loss function.

[0113] Specifically, in this embodiment, the Optical Flow Guided Optimization (OFGO) module includes the following implementations:

[0114] Adaptive Flow Weight (AFW) Calculation:

[0115] For each predicted displacement vector D j Calculate its corresponding weight:

[0116]

[0117] Where ∈ is a small constant used to ensure numerical stability.

[0118] AFW loss definition:

[0119] Calculate the weighted displacement prediction error:

[0120]

[0121] Among them G j It is the actual displacement vector, D j It is the predicted displacement vector.

[0122] Target detection loss:

[0123] Using the standard object detection loss function:

[0124]

[0125] in It is classification loss. It is a regression loss. This is the targeted loss. Total loss function:

[0126] Combining AFW loss and target detection loss:

[0127]

[0128] The training process in this embodiment

[0129] Data preparation:

[0130] Extract consecutive frame sequences from UAV video datasets.

[0131] Image preprocessing includes resizing and normalization.

[0132] Model initialization:

[0133] The backbone network is initialized using pre-trained YOLOX-L weights.

[0134] Randomly initialize the parameters of the FGDP and OFGO modules.

[0135] Forward propagation:

[0136] Input three consecutive frames of images I t-1 ,I t and I t+1 .

[0137] Features are extracted using a backbone network and FPN.

[0138] Optical flow was calculated using FastFlowNet.

[0139] Dynamic forecasting is performed using the FGDP module.

[0140] Calculate the prediction results.

[0141] Backpropagation:

[0142] Calculate total loss The model parameters are updated through backpropagation.

[0143] Optimization process:

[0144] Using the Adam optimizer, the initial learning rate is 0.001.

[0145] The learning rate will decrease by 0.1 after each epoch.

[0146] Train for 8 epochs or until the model converges.

[0147] verify:

[0148] Evaluate model performance on the validation set.

[0149] Evaluation was conducted using metrics such as SAP, off AP, and VsAP.

[0150] Reasoning process

[0151] Input processing:

[0152] Receives continuous video frames.

[0153] Feature extraction:

[0154] Multi-scale features are extracted using a trained YOLOX-L backbone network and FPN. Optical flow calculation:

[0155] FastFlowNet is used to calculate optical flow between consecutive frames.

[0156] Dynamic prediction:

[0157] The FGDP module is used to predict location using extracted features and optical flow information.

[0158] Post-processing:

[0159] Non-maximum suppression (NMS) is applied to remove duplicate detections.

[0160] Filter the detection results based on a predefined confidence threshold.

[0161] Output result:

[0162] Generate the final target detection results, including the object's location, category, and confidence level.

[0163] Implementation effect

[0164] This invention validates the effectiveness of the StreamFlow method through experiments on several challenging UAV video datasets. The main implementation results are as follows:

[0165] Improved detection accuracy:

[0166] On datasets such as UAVDT, AUAIR, UAV123, and VisDrone, StreamFlow outperforms existing methods in terms of streaming average precision (sAP).

[0167] Especially when dealing with fast-moving objects, StreamFlow shows a significant advantage, with a marked improvement in the VsAP metric.

[0168] Real-time performance improvements:

[0169] Compared to baseline methods, StreamFlow significantly reduces processing latency while maintaining high detection accuracy.

[0170] At typical UAV video frame rates (such as 30fps), StreamFlow can achieve real-time target detection and tracking.

[0171] Robustness enhancement:

[0172] By integrating optical flow information, StreamFlow performs more stably when handling challenging scenes such as complex backgrounds, occlusions, and fast camera movements.

[0173] The FGDP module effectively captures fine-grained motion information of objects, reducing target loss and false detections.

[0174] Generalization ability:

[0175] StreamFlow performs well on various types of UAV video datasets, indicating that the method has strong generalization ability.

[0176] Even in unseen UAV scenarios, StreamFlow maintains high detection performance.

[0177] Computational efficiency:

[0178] Despite the introduction of additional optical flow calculations and the FGDP module, StreamFlow maintains low computational complexity through optimized implementation.

[0179] On standard GPU hardware, StreamFlow can achieve real-time processing to meet the needs of practical applications.

[0180] The StreamFlow method proposed in this invention significantly improves the real-time target detection performance in UAV video by effectively integrating optical flow information. Key innovations include:

[0181] A Flow-Guided Dynamic Prediction (FGDP) module was designed to achieve accurate trajectory prediction and position refinement.

[0182] An Optical Flow Guided Optimization (OFGO) module was introduced to optimize the model parameter learning process.

[0183] An adaptive flow weight (AFW) mechanism is proposed to enhance the model's ability to learn reliable flow information.

[0184] Experiments on multiple UAV video datasets demonstrate that StreamFlow exhibits superior performance in detection accuracy, real-time performance, and robustness. This method provides a novel solution for real-time target detection in UAV videos, with broad application prospects, particularly suitable for real-time video stream processing tasks in fields such as drone surveillance and autonomous driving.

[0185] This embodiment presents a novel streaming sensing method called StreamFlow. StreamFlow enhances real-time target detection capabilities by integrating optical flow information, improving the prediction accuracy of object positions in UAV videos. The core innovation of this method lies in the introduction of two key modules: Flow-Guided Dynamic Prediction (FGDP) and Optical Flow Guided Optimization (OFGO).

[0186] The core content of the technical solution is as follows:

[0187] An optical flow-guided dynamic prediction method is adopted, which combines the appearance features of future frames to enhance the ability to locate and identify objects in consecutive frames.

[0188] A flow-guided dynamic prediction (FGDP) module was designed, which uses local optical flow information to refine the prediction results, thereby achieving accurate trajectory prediction and position refinement.

[0189] An Optical Flow Guided Optimization (OFGO) module was introduced, which combines Adaptive Flow Weight (AFW) loss and target detection loss to optimize model parameters.

[0190] Building upon StreamYOLO, this paper optimizes streaming sensing technology by integrating optical flow information, enabling better processing of spatiotemporal data in video streams.

[0191] This embodiment improves the detection accuracy and real-time performance of fast-moving objects in UAV videos.

[0192] By introducing FGDP and OFGO modules, reliable optical flow information extraction and utilization were achieved, enhancing the feature consistency between video frames.

[0193] Experiments on several challenging UAV video datasets demonstrate that StreamFlow outperforms existing methods in streaming perception tasks.

[0194] This technical solution has broad application prospects and is particularly suitable for real-time video streaming tasks, such as drone monitoring and autonomous driving.

[0195] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A real-time target detection streaming sensing method based on optical flow guidance, characterized in that, include: Extracting multi-scale features from two consecutive frames of images; Calculate the optical flow field of two consecutive frames; Based on the target detection results of the current frame, select the optical flow information of the neighborhood surrounding the target in the optical flow field; Location prediction is performed using the aforementioned multi-scale features and selected optical flow information; The predicted location is optimized to obtain the final target detection result; Extracting multi-scale features from two consecutive frames of images includes: Basic features of two consecutive frames of images were extracted using the YOLOX-L backbone network. Based on the aforementioned fundamental features, FPN is used to fuse features from different levels to generate multi-scale features. Calculating the optical flow field of two consecutive frames includes: Calculate the optical flow field of two consecutive frames using FastFlowNet; The optical flow field is normalized. Location prediction using the multi-scale features and selected optical flow information includes: A dynamic motion predictor is built based on a convolutional neural network; The multi-scale features and selected optical flow information are input into the dynamic motion predictor to obtain the predicted displacement vector of the target. Update the target position using the predicted displacement vector; The updated location is combined with the target localization information of the YOLOX-L backbone network to obtain the final predicted bounding box coordinates; Optimizing the predicted location includes: For the predicted displacement vector, calculate the corresponding weights; Based on the weights, the weighted displacement prediction error is obtained; The predicted position is optimized by using the displacement prediction error and the preset target detection loss function.

2. The real-time target detection streaming sensing method based on optical flow guidance according to claim 1, characterized in that, The corresponding weights are: in, As weight, for Predict the square of the displacement in the direction. The coordinates are in the horizontal direction. For the target index, It is a stable term.

3. The real-time target detection streaming sensing method based on optical flow guidance according to claim 2, characterized in that, The weighted displacement prediction error is: in, It is the actual displacement vector. It is the predicted displacement vector. For the weighted displacement prediction error, The target quantity.

4. The real-time target detection streaming sensing method based on optical flow guidance according to claim 1, characterized in that, The method for updating the target position using the predicted displacement vector is as follows: in, For the updated target location, The target position for the current frame. It is the predicted displacement vector.

Images