Real-time target detection streaming sensing method based on optical flow guidance

Abstract

The invention relates to a real-time target detection streaming sensing method based on optical flow guidance. The method comprises the following steps: extracting multi-scale features of two continuous frames of images; calculating an optical flow field of two continuous frames of images; according to a target detection result of the current frame, selecting optical flow information of a target surrounding neighborhood in the optical flow field; performing position prediction by using the multi-scale features and the selected optical flow information; and optimizing the predicted position to obtain a final target detection result. According to the invention, the detection precision and the real-time performance of the fast moving object in the UAV video are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a real-time target detection streaming perception method based on optical flow guidance. Background Art

[0002] Real-time object detection in UAV videos is an important and challenging task. Traditional object detection methods often have latency issues when dealing with fast-moving objects and dynamic scenes, making it difficult to meet real-time requirements. To address this problem, researchers have proposed the concept of streaming perception, which aims to predict the future position of an object by analyzing historical data, thereby achieving real-time updates of the object's state.

[0003] Existing streaming perception methods, such as Streamer, StreamYOLO, LongShortNet, and DAMO-StreamNet, have made some progress in reducing latency and improving detection accuracy. However, these methods still face challenges in dealing with fast-moving objects in UAV videos, especially in capturing fine-grained motion information between consecutive frames. For example:

[0004] StreamYOLO directly inputs two consecutive frames into the network, but lacks a dedicated mechanism to extract and process motion information.

[0005] Although DAMO-StreamNet adopts a dual-branch structure to extract semantic and temporal features, it still has shortcomings in capturing key motion clues, especially when facing complex trajectories.

[0006] These methods have difficulty accurately representing continuous motion between frames, resulting in inaccurate detection or object loss in UAV scenarios. This challenge is further exacerbated in UAV surveillance applications, where the appearance and position of objects may change subtly but significantly due to the wide camera field of view and rapidly changing flight paths. Summary of the invention

[0007] The purpose of the present invention is to propose a streaming perception method for real-time target detection based on optical flow guidance to solve the problems existing in the above-mentioned prior art: it is difficult to accurately capture fine-grained motion information between consecutive frames, the performance is poor when processing fast-moving objects and complex trajectories, and it is difficult to adapt to the subtle but significant changes in the appearance and position of objects in UAV scenes. By integrating optical flow information to enhance the real-time target detection capability, the prediction accuracy of the object position in the UAV video is improved.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A real-time target detection streaming perception method based on optical flow guidance, comprising:

[0010] Extract multi-scale features of two consecutive frames of images;

[0011] Calculate the optical flow field of two consecutive frames of images;

[0012] Selecting optical flow information of a neighborhood around the target in the optical flow field according to the target detection result of the current frame;

[0013] Performing position prediction using the multi-scale features and the selected optical flow information;

[0014] Optimize the predicted position to obtain the final target detection result.

[0015] Optionally, extracting multi-scale features of two consecutive frames of images includes:

[0016] The basic features of two consecutive frames of images are extracted through the YOLOX-L backbone network;

[0017] Based on the basic features, FPN is used to fuse features at different levels to generate multi-scale features.

[0018] Optionally, calculating the optical flow field of two consecutive frames of images includes:

[0019] Calculate the optical flow field of two consecutive frames of images through FastFlowNet;

[0020] Normalize the optical flow field.

[0021] Optionally, using the multi-scale features and the selected optical flow information to perform position prediction includes:

[0022] Building a dynamic motion predictor based on convolutional neural networks;

[0023] Inputting the multi-scale features and the selected optical flow information into a dynamic motion predictor to obtain a predicted displacement vector of the target;

[0024] updating a target position using the predicted displacement vector;

[0025] Combine the updated position with the target positioning information of the YOLOX-L backbone network to obtain the final predicted bounding box coordinates.

[0026] Optionally, optimizing the predicted position includes:

[0027] For the predicted displacement vector, calculating a corresponding weight;

[0028] Based on the weights, obtaining a weighted displacement prediction error;

[0029] The predicted position is optimized using the displacement prediction error and a preset target detection loss function.

[0030] Optionally, the corresponding weight is:

[0031]

[0032] Among them, w j is the weight, is the square of the predicted displacement in the x direction, x is the horizontal coordinate, j is the target index, and ∈ is the stability term.

[0033] Optionally, the weighted displacement prediction error is:

[0034]

[0035] Among them, G j is the real displacement vector, D j is the predicted displacement vector, is the weighted displacement prediction error, and M is the number of targets.

[0036] Optionally, the method of updating the target position using the predicted displacement vector is:

[0037]

[0038] in, is the updated target position, is the target position of the current frame, D j is the predicted displacement vector.

[0039] The beneficial effects of the present invention are:

[0040] The present invention predicts the position by utilizing the multi-scale features and the selected optical flow information; optimizes the predicted position to obtain the final target detection result; realizes reliable extraction and utilization of optical flow information, and enhances the feature consistency between video frames; improves the detection accuracy and real-time performance of fast-moving objects in UAV videos; experiments on multiple challenging UAV video datasets show that StreamFlow performs better than existing methods in streaming perception tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1A schematic diagram of a streaming perception method for real-time target detection based on optical flow guidance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The main components in this embodiment include:

[0046] Multi-scale feature extraction network: consists of feature pyramid network (FPN) and YOLOX-L backbone network.

[0047] Flow-Guided Dynamic Prediction (FGDP) module: Utilizes temporal optical flow data for object trajectory prediction.

[0048] Optical flow guided optimization (OFGO) module: optimizes network parameters by considering localization loss and optical flow estimation reliability.

[0049] Adaptive Stream Weight (AFW) module: adjusts the influence of stream samples during training according to their reliability.

[0050] Target detection branch: processes the fused features and outputs the target detection results.

[0051] Streaming perception branch: Use the FGDP module to refine the prediction results.

[0052] Optical flow estimation module: Calculates the optical flow field between consecutive frames.

[0053] Dynamic Motion Predictor (DMP): Uses selected features to generate offset predictions for refining initial object detection results.

[0054] The experimental environment of this embodiment is:

[0055] Hardware setup: A standard computer vision platform equipped with four NVIDIA RTX 3090 GPUs was used.

[0056] Software environment: implemented based on the deep learning framework PyTorch.

[0057] This example experiment selects four widely used UAV video target detection datasets:

[0058] UAVDT dataset: contains approximately 80,000 frames of images from more than 10 hours of urban UAV footage, with vehicle annotations and attributes such as weather conditions and flight altitude.

[0059] AUAIR dataset: contains more than 32,000 labeled frames from more than 2 hours of RGB video, providing annotations for 8 traffic-related object categories and metadata such as GPS coordinates, IMU sensor data, etc.

[0060] UAV123 dataset: contains 123 video sequences with more than 110,000 frames, designed for aerial tracking challenges, with fully annotated upright bounding boxes.

[0061] VisDrone dataset: contains 288 video clips with a total of 261,908 frames and 10,209 static images from 14 Chinese cities, including more than 2.6 million bounding boxes.

[0062] In order to comprehensively evaluate the prediction accuracy and real-time performance, this embodiment uses the following evaluation indicators:

[0063] Streaming Average Precision (sAP): Evaluates the accuracy and timeliness of predictions. Including different scales (sAP l ,sAP m ,sAP s ) and IoU threshold (sAP 50 ,sAP 75 ) sub-indicator.

[0064] Offline Average Precision (off AP): evaluates the object detection performance without considering streaming constraints.

[0065] Speed-aware streaming average precision (VsAP): evaluates the performance at different object speeds.

[0066] The experimental configuration of this embodiment is:

[0067] Model size: StreamFlow model based on YOLOX-L.

[0068] Learning rate: The initial learning rate is 0.001 and decays by 0.1 in each epoch.

[0069] Batch size: Each batch contains 16 samples.

[0070] Optimizer: Adopt Adam optimizer.

[0071] Number of training rounds: The model is trained for 8 epochs.

[0072] Other parameters: Keep the same as StreamYOLO.

[0073] The present embodiment proposes a real-time target detection streaming perception method based on optical flow guidance, including:

[0074] Extract multi-scale features of two consecutive frames of images;

[0075] Calculate the optical flow field of two consecutive frames of images;

[0076] Selecting optical flow information of a neighborhood around the target in the optical flow field according to the target detection result of the current frame;

[0077] Performing position prediction using the multi-scale features and the selected optical flow information;

[0078] Optimize the predicted position to obtain the final target detection result.

[0079] Furthermore, extracting multi-scale features of two consecutive frames of images includes:

[0080] The basic features of two consecutive frames of images are extracted through the YOLOX-L backbone network;

[0081] Based on the basic features, FPN is used to fuse features at different levels to generate multi-scale features.

[0082] Specifically, in this embodiment, multi-scale feature extraction includes:

[0083] Input two consecutive frames of image I t-1 and I t .

[0084] Basic features are extracted through the YOLOX-L backbone network.

[0085] Use FPN to fuse features at different levels and generate multi-scale feature maps Where L represents the number of layers of the feature pyramid.

[0086] Furthermore, calculating the optical flow field of two consecutive frames of images includes:

[0087] Calculate the optical flow field of two consecutive frames of images through FastFlowNet;

[0088] Normalize the optical flow field.

[0089] Specifically, in this embodiment, optical flow estimation includes:

[0090] Input two consecutive frames of image I t-1 and I t .

[0091] The optical flow field O is calculated by FastFlowNet. x ,O y ).

[0092] Normalize the optical flow field.

[0093] Further, using the multi-scale features and the selected optical flow information to perform position prediction includes:

[0094] Building a dynamic motion predictor based on convolutional neural networks;

[0095] Inputting the multi-scale features and the selected optical flow information into a dynamic motion predictor to obtain a predicted displacement vector of the target;

[0096] updating a target position using the predicted displacement vector;

[0097] Combine the updated position with the target positioning information of the YOLOX-L backbone network to obtain the final predicted bounding box coordinates.

[0098] Specifically, in this embodiment, the flow guided dynamic prediction (FGDP) module implements:

[0099] Optical flow area selection:

[0100] According to the target detection result of the current frame Where M is the number of detected objects.

[0101] For each detected target j, the optical flow information of the 3×3 neighborhood around it is selected in the optical flow field O.

[0102] Dynamic Motion Predictor (DMP):

[0103] Design a convolutional neural network C to predict the displacement vector.

[0104] The input of DMP includes multi-scale feature maps F and selected optical flow information O j .

[0105] For each target j, predict the displacement vector: D j =C(F,O j ; Θ D ), where Θ D is a learnable parameter of DMP.

[0106] Location refinement:

[0107] Update the target position using the predicted displacement vector:

[0108] Combine the updated position with the target positioning information of the YOLOX backbone network to obtain the final predicted bounding box coordinates.

[0109] Further, optimizing the predicted position includes:

[0110] For the predicted displacement vector, calculating a corresponding weight;

[0111] Based on the weights, obtaining a weighted displacement prediction error;

[0112] The predicted position is optimized using the displacement prediction error and a preset target detection loss function.

[0113] Specifically, in this embodiment, the optical flow guided optimization (OFGO) module implementation includes:

[0114] Adaptive Flow Weight (AFW) calculation:

[0115] For each predicted displacement vector D j , calculate its corresponding weight:

[0116]

[0117] where ∈ is a small constant used to ensure numerical stability.

[0118] AFW loss definition:

[0119] Calculate the weighted displacement prediction error:

[0120]

[0121] Among them G j is the real displacement vector, D j is the predicted displacement vector.

[0122] Object detection loss:

[0123] Use the standard object detection loss function:

[0124]

[0125] in is the classification loss, is the regression loss, Is the target loss. The total loss function:

[0126] Combining AFW loss and target detection loss:

[0127]

[0128] The training process of this embodiment

[0129] Data preparation:

[0130] Extracting continuous frame sequences from UAV video datasets.

[0131] Preprocess the images, including resizing and normalization.

[0132] Model initialization:

[0133] Initialize the backbone network using pre-trained YOLOX-L weights.

[0134] Randomly initialize the parameters of the FGDP and OFGO modules.

[0135] Forward propagation:

[0136] Input three consecutive frames of image I t-1 ,I t and I t+1 .

[0137] Features are extracted through backbone network and FPN.

[0138] Optical flow is calculated using FastFlowNet.

[0139] Apply FGDP module for dynamic forecasting.

[0140] Calculate the prediction results.

[0141] Backward Propagation:

[0142] Calculate total loss Update the model parameters via back-propagation.

[0143] Optimization process:

[0144] The Adam optimizer is used with an initial learning rate of 0.001.

[0145] The learning rate is decayed by 0.1 after each epoch.

[0146] Train for 8 epochs or until the model converges.

[0147] verify:

[0148] Evaluate model performance on the validation set.

[0149] Evaluation is performed using metrics such as sAP, off AP, and VsAP.

[0150] Reasoning process

[0151] Input processing:

[0152] Receive consecutive video frames.

[0153] Feature extraction:

[0154] Use the trained YOLOX-L backbone network and FPN to extract multi-scale features. Optical flow calculation:

[0155] FastFlowNet is used to compute the optical flow between consecutive frames.

[0156] Dynamic prediction:

[0157] The FGDP module is applied to perform position prediction using the extracted features and optical flow information.

[0158] Post-processing:

[0159] Non-maximum suppression (NMS) is applied to remove duplicate detections.

[0160] Filter detection results based on predefined confidence thresholds.

[0161] Output:

[0162] Generate the final target detection results, including the location, category and confidence of the object.

[0163] Implementation Effect

[0164] This paper verifies the effectiveness of the StreamFlow method by conducting experiments on multiple challenging UAV video datasets. The main implementation results are as follows:

[0165] Improved detection accuracy:

[0166] On datasets such as UAVDT, AUAIR, UAV123 and VisDrone, StreamFlow outperforms existing methods in the streaming average precision (sAP) indicator.

[0167] Especially when dealing with fast-moving objects, StreamFlow shows significant advantages and the VsAP indicator is significantly improved.

[0168] Real-time performance improvements:

[0169] Compared with the baseline method, StreamFlow significantly reduces the processing delay while maintaining high detection accuracy.

[0170] At typical UAV video frame rates (e.g., 30fps), StreamFlow is able to achieve real-time target detection and tracking.

[0171] Robustness enhancement:

[0172] By integrating optical flow information, StreamFlow performs more stably when dealing with challenging scenes such as complex backgrounds, occlusions, and fast camera motions.

[0173] The FGDP module effectively captures the fine-grained motion information of objects and reduces target loss and false detection.

[0174] Generalization ability:

[0175] StreamFlow performs well on different types of UAV video datasets, indicating that this method has strong generalization ability.

[0176] In unseen UAV scenarios, StreamFlow can still maintain high detection performance.

[0177] Computational efficiency:

[0178] Despite the introduction of additional optical flow calculation and FGDP modules, StreamFlow maintains low computational complexity through optimized implementation.

[0179] On standard GPU hardware, StreamFlow can achieve real-time processing to meet actual application needs.

[0180] The StreamFlow method proposed in this paper significantly improves the real-time target detection performance in UAV videos by effectively integrating optical flow information. The main innovations include:

[0181] A flow-guided dynamic prediction (FGDP) module is designed to achieve accurate trajectory prediction and position refinement.

[0182] The optical flow guided optimization (OFGO) module was introduced to optimize the model parameter learning process.

[0183] An adaptive flow weight (AFW) mechanism is proposed to enhance the model's ability to learn reliable flow information.

[0184] Through experimental verification on multiple UAV video datasets, StreamFlow has shown superiority in detection accuracy, real-time performance and robustness. This method provides a new solution for real-time target detection in UAV videos, which has broad application prospects and is particularly suitable for real-time video stream processing tasks in the fields of drone monitoring and autonomous driving.

[0185] This embodiment discloses a novel stream perception method named StreamFlow. StreamFlow enhances the real-time target detection capability by integrating optical flow information, and improves the prediction accuracy of the object position in the UAV video. The core innovation of this method is the introduction of two key modules: Flow-Guided Dynamic Prediction (FGDP) and Optical Flow Guided Optimization (OFGO).

[0186] The core contents of the technical solution are as follows:

[0187] The dynamic prediction method guided by optical flow is combined with the appearance features of future frames to enhance the ability to locate and recognize objects in consecutive frames.

[0188] A flow-guided dynamic prediction (FGDP) module is designed to use local optical flow information to refine the prediction results and achieve accurate trajectory prediction and position refinement.

[0189] The optical flow guided optimization (OFGO) module is introduced to optimize the model parameters by combining the adaptive flow weight (AFW) loss and the object detection loss.

[0190] Based on StreamYOLO, the streaming perception technology is optimized by integrating optical flow information to better process the spatiotemporal data in the video stream.

[0191] This embodiment improves the detection accuracy and real-time performance of fast-moving objects in UAV videos.

[0192] By introducing the FGDP and OFGO modules, reliable optical flow information extraction and utilization are achieved, and the feature consistency between video frames is enhanced.

[0193] Experiments on multiple challenging UAV video datasets demonstrate that StreamFlow outperforms existing methods in streaming perception tasks.

[0194] This technical solution has broad application prospects and is particularly suitable for real-time video stream processing tasks such as drone monitoring and autonomous driving.

[0195] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A real-time target detection streaming perception method based on optical flow guidance, characterized in that: include: Extract multi-scale features of two consecutive frames of images; Calculate the optical flow field of two consecutive frames of images; Selecting optical flow information of a neighborhood around the target in the optical flow field according to the target detection result of the current frame; Performing position prediction using the multi-scale features and the selected optical flow information; Optimize the predicted position to obtain the final target detection result.

2. The real-time target detection streaming perception method based on optical flow guidance according to claim 1 is characterized in that: Extracting multi-scale features of two consecutive frames of images includes: The basic features of two consecutive frames of images are extracted through the YOLOX-L backbone network; Based on the basic features, FPN is used to fuse features at different levels to generate multi-scale features.

3. The real-time target detection streaming perception method based on optical flow guidance according to claim 1 is characterized in that: Calculating the optical flow field of two consecutive frames of images includes: Calculate the optical flow field of two consecutive frames of images through FastFlowNet; Normalize the optical flow field.

4. The real-time target detection streaming perception method based on optical flow guidance according to claim 1 is characterized in that: Using the multi-scale features and the selected optical flow information to perform position prediction includes: Building a dynamic motion predictor based on convolutional neural networks; Inputting the multi-scale features and the selected optical flow information into a dynamic motion predictor to obtain a predicted displacement vector of the target; updating a target position using the predicted displacement vector; Combine the updated position with the target positioning information of the YOLOX-L backbone network to obtain the final predicted bounding box coordinates.

5. The real-time target detection streaming perception method based on optical flow guidance according to claim 4 is characterized in that: Optimizing the predicted position includes: For the predicted displacement vector, calculating a corresponding weight; Based on the weights, obtaining a weighted displacement prediction error; The predicted position is optimized using the displacement prediction error and a preset target detection loss function.

6. The real-time target detection streaming perception method based on optical flow guidance according to claim 5 is characterized in that: The corresponding weights are: Among them, w j is the weight, is the square of the predicted displacement in the x direction, x is the horizontal coordinate, j is the target index, and ∈ is the stability term.

7. The real-time target detection streaming perception method based on optical flow guidance according to claim 5 is characterized in that: The weighted displacement prediction error is: Among them, G j is the real displacement vector, D j is the predicted displacement vector, is the weighted displacement prediction error, and M is the number of targets.

8. The real-time target detection streaming perception method based on optical flow guidance according to claim 4 is characterized in that: The method of updating the target position using the predicted displacement vector is: in, is the updated target position, is the target position of the current frame, D j is the predicted displacement vector.

Images