Unmanned aerial vehicle small target trajectory repair method based on density adaptive slice and KF-LSTM residual error

Abstract

The invention discloses an unmanned aerial vehicle small target trajectory repair method based on density adaptive slicing and KF-LSTM residual errors. The method comprises the steps of video preprocessing, GPU parallel decoding of 4K video streams, denoising and color correction in combination with adaptive median filtering and a gray world algorithm, and dynamic downsampling when a load exceeds a limit. Performing dynamic slicing, identifying a target area through lightweight semantic segmentation, dividing image sub-blocks according to density, and generating an enhanced feature map through bilinear interpolation fusion; lightweight detection, channel pruning and hybrid quantization (INT8 / FP16) optimization of the convolutional network, target detection and low-confidence target marking; track repairing is carried out, a motion state vector is constructed, and after KF-LSTM residual error correction is carried out, three-order weighting verification is carried out through a color histogram, an optical flow trend and a semantic position; and correcting the confidence coefficient, fusing the multi-frame consistency score by the Bayesian model, and outputting a refinement track with the confidence coefficient greater than or equal to 0.8. According to the method, 4K and 30FPS processing is realized on an edge device less than or equal to 5TFLOPS, the small target detection precision is improved by more than one time, and the mistakenly abandoned target rescue rate gt; and the track smoothness is improved by 43.6%.

Description

TECHNICAL FIELD

[0001] The application belongs to the cross technical field of computer vision, edge computing and intelligent transportation, and specifically relates to a lightweight small target trajectory repair method for unmanned aerial vehicle high-altitude shooting scenes, which is mainly applied to:

[0002] (1) real-time detection of vehicle / pedestrian small targets (pixel ratio <0.01%) in unmanned aerial vehicle traffic monitoring;

[0003] (2) 4K video stream processing on edge devices (computing power ≤5TFLOPS);

[0004] (3) multi-frame joint decision and repair of low-confidence targets (confidence lower than a set threshold); and (4) trajectory prediction error compensation for nonlinear motion scenes such as unmanned aerial vehicle sharp turns and obstacle avoidance maneuvers.

[0005] The application solves the three technical bottlenecks of feature sparsity, high rejection rate and large computing power demand in small target detection in traditional solutions through the synergistic innovation of the spatial enhancement mechanism of dynamic slicing and the temporal verification mechanism of multi-frame Bayesian decision. BACKGROUND

[0006] The current unmanned aerial vehicle traffic monitoring technology faces three important technical problems: first, in the spatial dimension, high-altitude shooting results in a target pixel ratio generally lower than one ten-thousandth (for example, a 20x20px target in a 4K image accounts for only 0.0048%), and existing convolutional neural networks cannot extract effective features due to receptive field mismatch and information loss caused by downsampling.

[0007] Second, in the time dimension, a considerable proportion of low-confidence targets (0.1-0.5) are real targets, but traditional single-frame decision mechanisms discard them directly, and unmanned aerial vehicle jitter causes target trajectory breaks.

[0008] Third, in the computing dimension, 4K video real-time processing requires 200+TOPS computing power, which is far beyond the upper limit of the computing power of typical edge devices (such as the 1.3TFLOPS computing power of Jetson TX2).

[0009] These problems result in large small target flow statistical errors and decreased fog detection accuracy in actual applications. Although existing solutions attempt to improve through increasing resolution, improving network structure (such as feature pyramid optimization), and introducing filtering algorithms, none of them can simultaneously solve the precision and efficiency contradiction, and innovative technical breakthroughs are urgently needed. SUMMARY

[0010] The application proposes a "spatial-temporal dual-dimension enhancement" architecture that breaks the bottleneck of traditional target detection and tracking in complex scenes, optimizes feature capture from the spatial dimension, guarantees tracking continuity from the temporal dimension, and constructs a complete technical system with multiple modules cooperating.

[0011] To achieve the above, the application provides a UAV small target trajectory repair method with density adaptive slicing and KF-LSTM residual, and the specific steps are as follows:

[0012] S1: access the 4K resolution video input source, use the GPU parallel decoding technology to quickly decode the video stream, and obtain the video frame. Denoising and color correction are performed on the decoded video frame: adaptive median filtering is used to remove salt and pepper noise, and the color deviation caused by uneven lighting is corrected by the gray world algorithm, and the preprocessed standard frame image is output.

[0013] S2: a lightweight image semantic segmentation network is used to identify candidate small target regions in the scene, and the original image is dynamically divided into multiple scale-variable sub-blocks according to the target density. Perform bilinear interpolation on each sub-block to restore its original resolution, and use a lightweight feature extraction network to extract its key visual features. The sub-block features are spliced and fused to construct a global enhanced feature map containing small target high-density information, so as to improve the feature saliency and detection success rate of small targets.

[0014] S3: The feature map output by the dynamic slicing module is used to perform target detection by a lightweight convolutional neural network optimized by channel-level structured pruning and low-precision representation. The model outputs the target detection result, including target category, bounding box coordinates and initial confidence, and marks potential low-confidence targets (confidence <0.5) to enter the trajectory repair process.

[0015] S4: Construct the target motion state vector X t , predict the position X kf using Kalman filter theory, and correct the detection frame through affine transformation, calculate the residual R t between the current frame detection position and the predicted position; input the normalized residual sequence of the past 5 frames into the lightweight LSTM network (1 layer, 16 units), output the residual correction term ΔX; fuse to generate the final predicted state X fused = X kf + ΔX. Synchronously extract the HSV color histogram to calculate the Bhattacharyya distance and verify the color consistency, compare the optical flow field with the predicted motion vector to verify the trend matching degree, and verify the rationality of the target space position based on the scene semantic constraint. Whether to repair the trajectory is determined by a three-order verification weighted voting mechanism.

[0016] S5: For the repair target verified, perform pre-frame feature point matching and track merging processing to form a complete continuous target track. Combine the historical confidence distribution, color / motion / semantic consistency score to build a Bayesian correction model, and recalculate the target confidence. Output a high-confidence target set with a confidence of not less than 0.8, including target category, refined bounding box and continuous track information, supporting real-time display, storage and docking with upper-layer applications.

[0017] Further, in the step S1, comprising:

[0018] S101, first support RTSP, HDMI and local storage file and other multi-heterogeneous 4K video source access, call hardware acceleration decoding library such as NVIDIATMVideo Codec SDK, rely on GPU parallel computing to realize real-time video frame decoding;

[0019] S102, then adopt space-time domain joint processing, space domain uses non-local mean algorithm to suppress single-frame Gaussian noise, time domain eliminates motion noise through multi-frame motion compensation, combines Retinex theory to decompose illumination and reflection components, and enhances the gray difference of low-contrast scene targets;

[0020] S103, finally, real-time monitor the device load, dynamically downsample the video stream to 2K or 1080P by Lanczos interpolation for the super-threshold load, and unify the input frame rate to 30FPS through frame replication or optical flow interpolation to complete 4K video stream input and preprocessing.

[0021] Further, in the step S3, comprising:

[0022] S301, construct a channel importance evaluation system based on Taylor expansion, calculate the gradient sensitivity of each convolution layer channel, and prune the channels with sensitivity lower than the threshold. Through grouping convolution and residual connection to reconstruct the network structure, the feature expression ability is maintained while the model parameter quantity is reduced by more than 60%;

[0023] S302, adopt non-uniform quantization strategy, perform INT8 quantization on weight parameters, and use dynamic FP16 / INT8 mixed precision representation on activation values. Design quantization-aware training (QAT) process, insert pseudo-quantization nodes in the training process, minimize quantization loss through adversarial training, and ensure that the mAP (mean average precision) of the quantized model decreases by no more than 2%;

[0024] S303, for Jetson TX2 and other embedded devices, based on the inference engine of TensorRT, through layer fusion, memory reuse and other technologies, optimize the model calculation graph into an efficient execution flow. Combined with CUDA multi-thread parallel technology, realize real-time detection performance of more than 30FPS at 4K resolution;

[0025] S304, the model outputs the target category, the bounding box coordinates, and the initial confidence. After that, a dynamic confidence threshold adjustment mechanism based on scene complexity is constructed. When the target density exceeds the preset threshold, the screening threshold is automatically reduced to 0.45, and potential targets with an initial confidence of less than 0.5 are marked for secondary verification through a spatio-temporal context association algorithm.

[0026] The beneficial effects of the present application are:

[0027] 1. The small target detection accuracy is significantly improved (more than 80% higher than the traditional scheme) through the synergistic effect of dynamic slice local enhancement and multi-frame trajectory repair.

[0028] 2. 4K video high frame rate processing (≥30FPS) is realized on edge devices.

[0029] 3. Based on multi-frame Bayesian decision (color / motion / space three-order verification), the real targets that are mistakenly discarded are effectively rescued (rescue rate is more than 40%).

[0030] 4. LSTM correction Kalman filter residual error reduces position error in sharp turn scenes and improves non-linear motion prediction accuracy.

[0031] 5. Compared with the traditional scheme, the deployment cost is significantly reduced, the hardware cost is reduced, the model size is compressed, and the domestic edge computing platform is compatible (can run on edge devices with ≤5TFLOPS computing power). BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0033] Figure 1 : is the overall flowchart of the small target trajectory repair method of the unmanned aerial vehicle of the density adaptive slicing and KF-LSTM residual error of the present application.

[0034] Figure 2 : is a schematic diagram of the dynamic slicing module in the present application.

[0035] Figure 3 : is the working flowchart of the trajectory repairer proposed in the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0037] The application designs a density adaptive slicing and KF-LSTM residual unmanned aerial vehicle small target trajectory repair method, and the implementation process relies on an edge device (such as Jetson TX2) with an algorithm power ≤5TFLOPS, takes a 4K resolution (3840x2160px) and 30FPS video stream as input, realizes real-time detection and trajectory repair of small targets (pixel ratio <0.01%), and the specific implementation manner is as follows:

[0038] Step S1: 4K video stream access and preprocessing (corresponding to Figure 1 )

[0039] (1) Multi-source video input: supporting RTSP stream, HDMI input, and three types of data source access of local 4K video file. Calling NVIDIa Video Codec SDK hardware decoding library, through GPU parallel decoding (CUDA core number ≥256), single frame decoding delay ≤10ms, in H.265@60Mbps code stream, Jetson TX2 platform real-time decoding delay 8.3±1.2ms.

[0040] (2) Spatio-temporal joint denoising and enhancement: in the spatial domain, an adaptive median filter (window size 3x3 to 7x7 dynamically adjusted) is used to eliminate salt and pepper noise; in the time domain, motion compensation (optical flow method compensation displacement) based on 5 consecutive frames is used to suppress motion blur; in color correction, the gray world algorithm Wherein (gain coefficient α=1.2, β=0.9, γ=1.1), balance the RGB channel light deviation.

[0041] (3) Load adaptive downsampling: real-time monitoring of GPU utilization, when the load ≥80%: through Lanczos interpolation, 4K frame is downsampled to 2K (1920x1080px); frame replication technology (static scene) or TV-L1 optical flow interpolation (dynamic scene) is used to maintain the output frame rate of 30FPS.

[0042] Step S2: Dynamic slicing and feature enhancement (corresponding to Figure 2 )

[0043] (1) Lightweight semantic segmentation: deploying a SegNet model (parameter amount <1M) with MobileNetV3-Small as the backbone to segment the road, vehicle, and pedestrian candidate area. Output binary mask image, only when the target connected region pixel area ≥400px 2(1) Candidate target labeling: when the target size is less than 20x20px, mark it as a candidate target, ensuring the minimum effective feature area covering small targets (2) Dynamic slice generation: triggered when the target density distribution is greater than 10 targets per unit area: divide the original image into N x M sub-patches (N, M ∈ [2, 4]), and the sub-patch size is adaptive (the minimum is 640 x 480px); perform bilinear interpolation on each sub-patch to restore the original resolution (3840 x 2160px).

[0044] (3) Feature extraction and fusion: use a lightweight feature extraction network (ShuffleNetV2, depth multiplier 0.5x) to process the sub-patch; use channel attention mechanism (SE module) to weight and fuse the sub-patch features to generate a globally enhanced feature map (size 1920 x 1080 x 64).

[0045] Step S3: Lightweight target detection and low confidence screening (corresponding Figure 3 )

[0046] (1) Model compression and acceleration: channel pruning is based on Taylor expansion gradient sensitivity evaluation (threshold θ = 0.001), pruning redundant channels, model volume compression 62.3%;

[0047] Mixed precision quantization is weight INT8 quantization, and activation value FP16 dynamic conversion, quantization loss compensation factor λ = 0.2; TensorRT deployment: through layer fusion (Conv + BN + ReLU) and memory reuse, inference delay ≤ 15ms / frame.

[0048] (2) Dynamic confidence adjustment: the initial detection threshold is set to 0.5, and the confidence ≥ 0.5 is determined as an effective target. When the target density is greater than 10 per frame, the detection threshold is reduced to 0.45, and the confidence ≥ 0.45 is determined as a candidate target. At the same time, candidate targets with confidence ∈ [0.45, 0.5) are marked as low confidence targets and sent to the trajectory repair module for multi-frame verification.

[0049] Step S4: Multi-frame trajectory repair and verification (corresponding Figure 3 )

[0050] Motion state modeling: establish the state vector of the target with position, velocity, acceleration, rotation angle, width and height. State vector X t = [x, y, v x , v y , a x , a y , θ, w, h] T ; use Kalman filter to output the predicted state X kf , calculate the residual R t = X detected,t - X kfCache history 5 frames residual sequence, normalized input LSTM network, output residual correction term ΔX, inverse normalization ΔX denorm , generate fusion state X fused = X kf + ΔX denorm Error covariance matrix correction detection frame.

[0051] (2) Third-order verification mechanism (weighted voting weight): color consistency: calculate HSV histogram Bhattacharyya distance D Bhatt ≤0.3; Motion trend matching: contrast flow field and predicted vector angle θ ≤15°; Semantic space constraint: target needs to be in the road semantic segmentation area (IoU ≥0.7).

[0052] (3) Trajectory repair judgment: when the total score S total = w1S color + w2S motion + w3S semantic ≥7 (full score 10), it is judged as a real target.

[0053] Step S5: Bayesian confidence correction and output (corresponding Figure 3 )

[0054] (1) Trajectory merging: perform SURF feature point matching on the repaired target (match points ≥5 pairs), connect the broken trajectory; apply RANSAC filtering to remove abnormal points, and the trajectory smoothness is improved by 43.6%.

[0055] (2) Bayesian correction model:

[0056]

[0057] Where C i is the color / motion / semantic consistency score, P(T) is the prior probability of trajectory continuity (take 0.85). Output high confidence target (P new ≥0.8), label refined bounding box (Affine transformation correction) and continuous trajectory ID.

Claims

1. A method for UAV small target trajectory repair using density-adaptive slicing and KF-LSTM residuals, wherein the small target refers to a target with a pixel area ratio of less than 0.01% at 4K resolution, comprising the following steps: S1: Perform GPU parallel decoding on the input 4K video stream, remove salt-and-pepper noise through adaptive median filtering, correct color deviation by combining grayscale world algorithm, and dynamically downsample to 2K or 1080P based on device load. S2: A lightweight semantic segmentation network is used to identify candidate target regions, wherein the pixel area of ​​the candidate target region is ≥400px. 2 The original image is dynamically divided into variable-scale sub-patterns based on the target density. Bilinear interpolation is performed on each sub-pattern to restore the original resolution. After feature extraction, the sub-patterns are fused to generate a global enhanced feature map. S3: Detects targets using a lightweight convolutional network with channel pruning and mixed precision quantization, outputs target category, bounding box, and confidence score, and labels potential targets with confidence scores below a dynamic threshold; S4: Construct the target's motion state vector, predict its position by fusing Kalman filtering and LSTM, and predict the current state X based on historical states using Kalman filtering. kf Calculate the residual R t =X detected,t -X kf It also caches the residual sequence of historical k frames, inputs the normalized residual sequence into the LSTM network to output the residual correction term ΔX, and generates the fusion state X. fused =X kf +ΔX; S5: Based on X fused Then, three-level verifications were performed sequentially: color consistency (HSV histogram Bach distance ≤ 0.3), motion trend matching (optical flow angle ≤ 15°), and semantic space constraints (road area IoU ≥ 0.7), followed by weighted voting decision-making trajectory repair. S6: Perform multi-frame feature point matching and trajectory merging on the repair target, and fuse historical confidence and multi-dimensional consistency scores based on a Bayesian model to output a refined target trajectory with a confidence score ≥ 0.

8.

2. The method as described in claim 1, characterized in that, The dynamic downsampling in S1 specifically includes: real-time monitoring of GPU utilization, and downsampling 4K frames to 2K or 1080P through Lanczos interpolation when the load is ≥80%; using frame duplication for static scenes and TV-L1 optical flow interpolation for dynamic scenes to maintain 30FPS output.

3. The method as described in claim 1, characterized in that, The dynamic slicing generation of S2 satisfies the following: when the number of targets per unit area is greater than 10, the image is divided into N×M sub-blocks (N,M∈[2,4]), and the minimum sub-block size is 640×480px; the sub-block features are weighted and fused through the channel attention mechanism to generate an enhanced feature map of 1920×1080×64.

4. The method as described in claim 1, characterized in that, S3's lightweight detection includes: Taylor expansion-based gradient sensitivity evaluation pruning of redundant channels (threshold θ = 0.001), model volume compression ≥ 62.3%; weights are quantized using INT8, activation values ​​are dynamically switched between FP16 and INT8, and quantization loss compensation factor λ = 0.2; when the target density is > 10 / frame, the confidence threshold is dynamically reduced from 0.5 to 0.

45.

5. The method as described in claim 1, characterized in that, The third-order verification of S4 satisfies: the weighted total score S total =w1S color +w2S motion +w3S semantic Repair is triggered when the score is ≥7 (out of 10); the motion state vector is defined as X. t =[x,y,v x ,v y ,a x ,a y ,θ,w,h] T , 6. The method as described in claim 1, characterized in that, In S4, the LSTM residual correction satisfies the following conditions: the LSTM network structure is a 1-layer, 16-unit network, the input is a sequence of 5 historical residuals (9 dimensions), and the loss function during training is... When |ΔX|2>50px, switch back to pure Kalman filter output.

7. The method as described in claim 1, characterized in that, The Bayesian correction model for S5 is as follows: Where C i The color / motion / semantic consistency score is given, and P(T) is the prior probability of trajectory continuity.

8. A method for repairing the trajectory of a small UAV target using density-adaptive slicing and KF-LSTM residuals according to any one of claims 1-7, characterized in that, include: Video preprocessing module: Supports RTSP / HDMI / local 4K video source access, and integrates hardware accelerated decoding and load adaptive downsampling; Dynamic Slicing Module: A semantic segmentation network with MobileNetV3-Small backbone and ShuffleNetV2 feature extractor; Lightweight detection module: Based on the pruned quantization model, it supports dynamic confidence threshold adjustment; Trajectory repairer: Integrates Kalman filter prediction and third-order verification mechanism; Confidence correction module: Performs SURF feature point matching and Bayesian probability fusion.

Images