Detection method of moving targets in dynamic flight of UAV

Abstract

The present invention relates to the field of target detection and tracking technology, and in particular to a method for detecting moving targets during dynamic flight of unmanned aerial vehicles (UAVs). The method comprises: generating a vehicle model through a parameterized generation method to construct a vehicle dataset; constructing a simulated terrain using terrain elevation data; running the vehicle model on the simulated terrain at various speeds to generate a sequence of images containing various motion speeds, random camera shake, and different lighting conditions, thereby constructing a target motion dataset; extracting the target motion trajectory from each set of sequential images using a motion trajectory extraction method using multi-interval optical flow fusion, and pairing the images with the images from the previous step; constructing a deep learning network based on a dual-stream spatiotemporal attention mechanism; inputting the paired training dataset, performing network training, and storing a weight file; and, during an actual flight mission, calling the weight file to identify moving targets. The method has the advantages of achieving high accuracy, improved detection speed, and enhanced robustness through multimodal data processing, fusion, and optimization strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and in particular to a method for detecting a moving target in dynamic flight of an unmanned aerial vehicle (UAV). Background Art

[0002] With the rapid development of drone technology, drones are increasingly being used in military and civilian applications. Detecting and identifying moving targets on the ground or in the air is a critical task during drone flight. However, due to the numerous factors affecting drone flight, such as camera shake, lighting variations, and target speed changes, detecting and identifying moving targets becomes extremely challenging.

[0003] Traditional moving object detection methods are primarily based on optical flow or differential methods. While these methods achieve good results in simple scenarios, they often struggle in complex scenarios, such as those with dramatic lighting changes, rapid target motion, and significant camera shake. In recent years, deep learning technology has achieved remarkable results in the field of computer vision, and deep learning-based moving object detection methods have gradually become a research hotspot. However, existing deep learning methods mostly rely on large amounts of annotated data and still have shortcomings in multimodal data fusion and adaptability to dynamic environments. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method for detecting moving targets in the dynamic flight of an unmanned aerial vehicle.

[0005] The present invention aims to provide a method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle, which specifically comprises the following steps:

[0006] S1. Generate vehicle models of different sizes and types through a parametric generation method, build a parametric model library containing typical vehicles, generate multi-spectral vehicle model data, and construct a vehicle dataset;

[0007] S2. Constructing simulated terrain using terrain elevation data;

[0008] S3 target motion data set construction: the vehicle model generated in step S1 runs at various speeds on the simulated terrain constructed in step S2, generating a sequence of images containing various motion speeds, random lens shakes and different lighting conditions, and constructing a target motion data set;

[0009] S4. Extract the target motion trajectory of each set of sequence images in step S3 using the motion trajectory extraction method of multi-interval optical flow fusion, and pair it with the image data in step S3 to form a training data set;

[0010] S5. Build a deep learning network based on the dual-stream spatiotemporal attention mechanism; input the training dataset obtained by pairing in step S4, use the dynamic curriculum learning training method to train the network, and store the weight file after training;

[0011] S6. In actual flight missions, call the weight file to identify moving targets.

[0012] Preferably, step S1 specifically includes the following sub-steps:

[0013] S101. Define the vehicle length, width and height parameters as:

[0014] ;

[0015] Among them, the vehicle length L conforms to the uniform distribution with parameters lmin~lmax, lmin is the minimum length of a typical vehicle, and lmax is the maximum length of a typical vehicle; U is a uniform distribution, N is a Gaussian distribution, ΔW and ΔH To simulate the dimensional deviation of the actual vehicle;

[0016] S102. Generate multi-spectral vehicle model data for each generated vehicle model, including visible light image data and infrared image data; construct a vehicle dataset using the generated multi-spectral vehicle model data;

[0017] The method for generating visible light image data includes: simulating the optical reflection characteristics of the vehicle surface according to an optical reflection amount formula to generate visible light image data; the optical reflection amount formula is expressed as follows:

[0018] ;

[0019] in: represents the reflection function; i i represents the angle of incidence; i o represents the exit angle; i h It represents the angle between the half angle and the surface normal; α represents the surface roughness parameter; F 0 represents the value of Fresnel reflectivity when the incident angle is 0; i d It represents the angle of incidence i i and the exit angle i o The difference between

[0020] The method for generating infrared image data comprises: establishing a temperature gradient model between the engine area and the non-engine area, generating a temperature distribution on the vehicle surface according to the temperature gradient model; and generating infrared image data according to the temperature distribution.

[0021] The temperature distribution on the vehicle surface is expressed as follows:

[0022] ;

[0023] Where: T(x,y) is the temperature at position (x,y); Tcore is the engine temperature; d(x,y) is the distance from position (x,y) to the center of the engine, σ is the temperature attenuation coefficient; L is the vehicle length; Tenv is the ambient temperature.

[0024] Preferably, step S2 specifically includes the following sub-steps:

[0025] S201. Initialize parameters, including terrain roughness parameter H and number of iterations n;

[0026] S202. Generate multi-scale noise data using Perlin noise function; for each iteration k , calculate the noise function noise(2 k x ,2 k y ), and perform weighted superposition according to the formula;

[0027] S203. Calculate terrain elevation: For each location ( x , y ), calculate the elevation value according to the following formula H ( x , y ):

[0028] ;

[0029] Where: H ( x , y ) means at position ( x , y )’s terrain elevation value; n Indicates the number of iterations, that is, the number of layers of noise function calculation; k Indicates the index of the current iteration; 2 -kH represents the weight of each iteration;

[0030] S204. Use the generated terrain elevation data to construct a simulated terrain.

[0031] Preferably, step S3 specifically includes the following sub-steps:

[0032] S301 defines the vehicle motion trajectory, simulates the vehicle's posture changes on complex terrain, and constructs a vehicle motion trajectory model;

[0033] S302 simulates the translational shaking and rotational shaking of the UAV during flight, and establishes a translational shaking model and a rotational shaking model;

[0034] S303. The illumination angle is incremented by 5 minutes, simulating the solar azimuth and solar altitude at different time periods to establish an illumination angle progression model;

[0035] S304. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate 500 sets of visible light images and infrared images, each set including 100 frames of sequential images, to form a multi-spectral image dataset.

[0036] Preferably, step S301 specifically includes:

[0037] S3011. Define vehicle trajectory: The vehicle moves at a speed of v Movement along the terrain surface, position over time t The change is expressed as:

[0038] x vehicle( t )= x 0+ v · t

[0039] y vehicle( t )= y 0;

[0040] in: x 0. y 0 means the vehicle is x Axis and y The initial position of the axis; y 0 is fixed to the terrain center line; v The unit is pixels / second;

[0041] S3012. Based on terrain elevation H ( x , y ) Calculate the vehicle's roll and pitch angles to simulate the vehicle's posture changes on complex terrain;

[0042] The step S304 specifically includes:

[0043] S3041. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate a visible light image and an infrared image for each frame; render the generated visible light image and infrared image; the rendering method specifically includes:

[0044] For each set of data g and each frame k Perform the following operations:

[0045] Calculation time t = k ·Δ t Where, Δ t Express the frame interval;

[0046] according to x vehicle( t ) and terrain elevation H ( x , y ), adjust the vehicle posture;

[0047] Overlay camera translation shake offset (Δ x cam ( k ),Δ y cam ( k )) and camera rotation angle i cam ( k );

[0048] Render visible light and infrared images;

[0049] S3042. Apply affine transformation to the rendered image to simulate camera shake effect:

[0050] ;

[0051] Where: i cam is the rotation angle caused by camera shake; Δ x cam and Δ y cam is the translation offset caused by camera shake;

[0052] S3043. After rendering, 500 groups of visible light images and infrared images are generated, each group including 100 frames of sequential images, to form a multi-spectral image dataset.

[0053] Preferably, step S4 specifically includes the following sub-steps:

[0054] S401. Input data preprocessing: Input the 100-frame sequence image generated in step S3, and fuse the visible light image and infrared image to generate a multi-spectral grayscale image;

[0055] S402. Using the data pre-processed in step S401, multi-interval optical flow calculation is performed; multi-interval optical flow calculation includes short interval, medium interval and long interval optical flow calculation;

[0056] The specific calculation formula of the short-interval optical flow field is as follows:

[0057] ;

[0058] in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, window represents the window size;

[0059] The specific calculation formula of the intermediate optical flow field is as follows:

[0060] ;

[0061] in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image; poly_n=7 represents the polynomial kernel size;

[0062] ;

[0063] in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, edge_th=0.1 is the edge threshold;

[0064] S403. Dynamic weight allocation: Adaptively adjust the fusion weight based on the target area speed to optimize the tracking effect of targets with different speeds. The fusion formula is as follows:

[0065] ;

[0066] in, is the fused optical flow field; α 、 β 、 c represents weight;

[0067] S404. Trajectory extraction: Based on the fused optical flow field , update the target position corresponding to each frame image P t , calculate the target motion trajectory frame by frame;

[0068] S405. Trajectory pairing: Pair the target motion trajectory obtained in step S404 with the images in the target motion dataset in step S3, find the corresponding motion trajectory data for each frame of image, and form a training dataset.

[0069] Preferably, step S401 specifically includes:

[0070] S4011. Convert the visible light image into a single-channel grayscale image; the conversion formula is:

[0071] Gray rgb =0.299×R+0.587×G+0.114×B;

[0072] Where R, G, and B represent the pixel values ​​of the red, green, and blue channels respectively;

[0073] S4012. Normalize the infrared image and adjust the pixel value range to 0 to 255. The normalization formula is:

[0074] ;

[0075] Of which: IR min and IR max are the minimum pixel value and maximum pixel value of the current infrared image respectively;

[0076] S4013. Perform weighted fusion of the grayscale image and the normalized infrared image to generate a multi-spectral grayscale image. The fusion formula is:

[0077] Gray fused =α×Gray rgb +(1-α)×IR norm ;

[0078] Among them, Gray fused represents the fused multi-spectral grayscale image, and α represents the fusion weight coefficient;

[0079] S4014. Pyramid construction: using the fused multi-spectral grayscale image Gray fused Based on the image pyramid, a 4-layer image pyramid is created, and the top layer resolution is reduced to 240×135 by pyramid downsampling.

[0080] Preferably, the deep learning network based on the dual-stream spatiotemporal attention mechanism in step S5 includes a multi-spectral feature extraction branch, a motion trajectory embedding branch and a cross-modal fusion module;

[0081] The network structure of the multi-spectral feature extraction branch adopts an improved ResNet-50 backbone network and adds a spectral attention module to enhance the representation capability of multi-spectral features. The first-layer convolution kernel is adjusted to 7×7×4. The input is an RGB-IR four-channel spliced ​​image. The spectral attention module outputs a weighted feature map SA(F). The feature representation capability is enhanced using the following calculation formula:

[0082] SA(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) F;

[0083] Where: SA(F) represents the output of the spectral attention module; F represents the feature map; σ is the sigmoid activation function; MLP represents the multi-layer perceptron; AvgPool(F) represents the global average pooling of the feature map F; MaxPool(F) represents the global maximum pooling of the feature map F; Represents element-wise multiplication;

[0084] The motion trajectory embedding branch uses a 3D convolutional temporal encoder, which consists of four 3×3×3 convolutional layers. The spatiotemporal position encoding is implemented by the following formula:

[0085] ;

[0086] Where: pos is the position index of the element in the sequence; i is the dimension index, starting from 0; d model is the dimension of the position encoding vector;

[0087] The cross-modal fusion module uses a gated cross-attention mechanism to dynamically fuse spectral and motion features. The calculation process is as follows:

[0088] Q=LN(F spectral )

[0089] K=V=LN(F motion )

[0090] GatedAttn=Softmax(QK T / )⊙Sigmoid(Conv(Q))

[0091] F fusion =GatedAttn V;

[0092] Where: Q, K, V represent query, key, and value respectively; LN represents layer normalization; F spectral represents the feature map obtained from the multi-spectral feature extraction branch; Fmotion represents the feature map obtained from the motion trajectory embedding branch; QK T Indicates Q and K T The dot product of represents the scaling factor of the dimension; ⊙ is the Hadamard product; F fusion Represents the fused feature map, the output obtained by the cross-modal fusion module; Represents a tensor product.

[0093] Preferably, step S5 specifically includes:

[0094] S501. Multi-spectral input data stitching: Perform affine transformation registration on the visible light image and infrared image at the same time. Then stitch the registered infrared channel and visible light channel along the channel dimension to form a 4-channel input with a size of 256×256×4.

[0095] S502. Construct the deep learning network based on the dual-stream spatiotemporal attention mechanism;

[0096] S503. Network training is performed using a dynamic curriculum learning training method, including a basic training phase, a multimodal fusion phase, and a robustness enhancement phase. During the training process, a multi-task joint training strategy is used to optimize network performance. After training is completed, the optimal weight file of the network is exported and the network structure and parameter configuration are stored.

[0097] In the basic training phase, 128×128 resolution visible light images are used for preliminary training to learn basic feature representations; a cosine decay learning rate schedule is used, with an initial learning rate of 1e-3;

[0098] In the multimodal fusion stage, the input image resolution is increased to 256×256, and infrared images and optical flow data are gradually introduced to enable the network to learn more complex multimodal features; the learning rate is reset to 5e-4;

[0099] The robustness enhancement stage enhances the robustness of the network by adding random motion blur and simulating extreme lighting changes;

[0100] The multi-task joint training strategy uses a composite loss function to optimize network performance; the composite loss function is defined as: L = l 1 L det + l 2 L flow + l 3 L reg ;

[0101] Where, L detrepresents the target detection loss; L flow represents the optical flow regression loss; L reg Indicates motion consistency loss; l 1 、l 2 、l 3 represents hyperparameters;

[0102] Object detection loss L det The improved Focal Loss loss function is used to adjust the parameter factor c =3.0, weight α =0.8;

[0103] Optical flow regression loss L flow Using Charbonnier loss:

[0104] ;

[0105] Where, x and y are the predicted value and the true value, and the parameter =1 e -3;

[0106] Motion consistency loss L reg As a trajectory smoothness constraint, the calculation formula is:

[0107] ;

[0108] Where, p t It's in time t The predicted position of Δ p t It's in time t The true displacement of Represents the Euclidean norm, which is used to calculate the distance between vectors;

[0109] After training is completed, the optimal weight file is exported to store the network structure and parameter configuration.

[0110] Preferably, step S6 specifically includes:

[0111] S601. During the flight mission, use a visible light camera and an infrared camera to simultaneously capture visible light and infrared images of the target. Use a SURF feature detector to identify key feature points in the image. Perform affine transformation registration based on the detected feature points and calculate the affine transformation matrix T:

[0112] T=estimateAffine2D(rgbkpts ,ir kpts );

[0113] ir aligned =warpAffine(ir img ,T,(width,height));

[0114] Where: rgb kpts and ir kpts Represent the key feature points of the visible light image and infrared image respectively; the input of the estimateAffine2D algorithm is two sets of corresponding feature points, and the output is the 2D affine transformation matrix T; warpAffine represents the image transformation function; (width, height) represents the size of the transformed image;

[0115] S602. Motion compensation is performed based on the inertial measurement unit data, and the aligned images are corrected by applying a homography transformation; Kalman filtering is used to process the de-jittered images and send them to the online inference optimization stage;

[0116] S603. Enable mixed precision computing when deploying embedded boards; dynamically adjust the resolution of the de-jittered image based on the flight altitude to obtain an optimized image; the optimized image data is fed into the motion trajectory prediction module, which predicts the position of the next three frames based on the long short-term memory network prediction unit integrated within the module;

[0117] S604. Result post-processing and fusion: A weighted box fusion algorithm is used to aggregate detection results of different scales to improve detection accuracy and robustness. The fusion formula is as follows:

[0118] fused box =∑(w i box i ) / ∑w i

[0119] w i =confidence i ×IoU i

[0120] In the formula: fused box represents the bounding box after fusion, w i is the weight of the detection box, box i Represents the i-th detection box, confidence i Represents confidence, IoU i represents the intersection and union ratio;

[0121] Compare the intersection-over-union (IoU) of the visible light image and the infrared image. If the IoU is less than 0.3, call the reject_low_confidence() function to reject the low-confidence detection result; generate a visualization result with trajectory prediction.

[0122] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0123] (1) Improve detection accuracy: By adopting the multi-scale detection aggregation (WBF) algorithm, this technical solution can integrate detection results of different scales, thereby significantly improving the accuracy of target detection; the weighted box fusion (WBF) algorithm optimizes the selection of detection boxes by considering confidence and intersection over union (IoU), reducing false detections and missed detections.

[0124] (2) Enhanced robustness: By comparing the intersection-over-union (IoU) of visible light (RGB) and infrared (IR) detection results through a cross-modal verification mechanism, the consistency of the detection results is ensured and the robustness of the system is enhanced. By dynamically adjusting the resolution and introducing random motion blur and illumination changes, the system can maintain stable performance under different environmental conditions.

[0125] (3) Improved real-time performance: Real-time data preprocessing includes multi-spectral alignment and dynamic de-jittering, ensuring high-quality input of image data; online inference optimization improves processing speed and prediction accuracy through adaptive computing scheduling and motion trajectory prediction modules, achieving real-time visualization output at 30fps.

[0126] (4) Resource optimization: By enabling mixed-precision computing when deploying embedded boards, computing efficiency is significantly improved while reducing precision, and hardware resource consumption is reduced.

[0127] (5) Strong adaptability: The dynamic curriculum learning training method allows the model to learn gradually at different stages, from basic features to multimodal features, and then to robustness enhancement, enabling the model to adapt to complex multimodal data.

[0128] In summary, the present invention provides a method for detecting moving targets in dynamic flight of unmanned aerial vehicles. Through innovative multimodal data processing, fusion and optimization strategies, it achieves improved detection speed and robustness while maintaining high accuracy, and is suitable for resource-constrained embedded environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0129] Figure 1 The present invention provides a flow chart of a method for detecting a moving target in a dynamic flight of a UAV.

[0130] Figure 2 This is a deep learning network framework diagram based on a dual-stream spatiotemporal attention mechanism provided according to an embodiment of the present invention.

[0131] Figure 3 4 is an architecture diagram of a multi-spectral feature extraction branch provided according to an embodiment of the present invention.

[0132] Figure 4 This is the detailed structure of the spectral attention module (SAM) provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0133] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, identical modules are denoted by identical reference numerals. In the case of identical reference numerals, their names and functions are also identical. Therefore, their detailed description will not be repeated.

[0134] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0135] The present invention provides a method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle, which specifically comprises the following steps:

[0136] S1. Generate vehicle models of different sizes and types using a parametric generation method to build a parametric model library containing typical vehicles; generate multi-spectral vehicle model data for each generated vehicle model, and construct a vehicle dataset using the generated multi-spectral vehicle model data; specifically, the following sub-steps are included:

[0137] S101. Define the vehicle length, width and height parameters as:

[0138] ;

[0139] Among them, the vehicle length L conforms to the uniform distribution with parameters lmin~lmax, lmin is the minimum length of a typical vehicle, and lmax is the maximum length of a typical vehicle; U is a uniform distribution, N is a Gaussian distribution, ΔW and ΔH To simulate the dimensional deviation of the actual vehicle;

[0140] Generate vehicle models of different sizes and types based on parameters, and build a parameterized model library containing typical vehicles. By adjusting these parameters, the diversity of the dataset can be enriched and the generalization ability of the model can be improved.

[0141] Brief Principle: Parametric generation is a technique for generating complex objects using mathematical models and parameters. Vehicle models can be parameterized using the following methods: (1) Geometric parameters: length, width, height, track width, wheelbase, etc.; (2) Appearance parameters: color, material, texture, etc.; (3) Physical parameters: mass, moment of inertia, powertrain parameters, etc.; (4) Motion parameters: speed, acceleration, steering angle, etc. By adjusting these parameters, different types of vehicle models can be generated.

[0142] S102. Generate multispectral vehicle model data for each generated vehicle model, including visible light (RGB) image data and infrared (IR) image data; construct a vehicle dataset using the generated multispectral vehicle model data. This step generates a visible light (RGB) and infrared (IR) multispectral dataset by combining optical and thermal radiation property modeling.

[0143] Specifically, the method for generating visible light image data includes: simulating the optical reflection characteristics of the vehicle surface according to an optical reflection amount formula to generate realistic visible light image data; the optical reflection amount formula is expressed as follows:

[0144] ;

[0145] in: Represents the reflection function, which is used to describe the direction of incident light oh i and the direction of the emitted light oh o The reflective relationship between i i represents the angle of incidence; i o represents the exit angle; i h It represents the angle between the half angle and the surface normal; α represents the surface roughness parameter; F 0 represents the value of Fresnel reflectivity when the incident angle is 0; i d It represents the angle of incidence i i and the exit angle i o The difference between i d = i i - i o ; Related angles i d It plays a key role in simulating the optical reflective properties of vehicle surfaces. It can accurately describe the relationship between the incident and outgoing light directions, thereby affecting the intensity and direction of the reflected light.

[0146] The above formula combines the effects of the normal distribution function, Fresnel term, and geometric occlusion function to calculate the reflection amount under specified incident and outgoing light directions.

[0147] Specifically, the method for generating infrared image data includes: establishing a temperature gradient model between the engine area and the non-engine area, generating a temperature distribution on the vehicle surface according to the temperature gradient model; generating infrared image data according to the temperature distribution;

[0148] The temperature distribution on the vehicle surface is expressed as follows:

[0149] ;

[0150] Where: T(x,y) is the temperature at position (x,y); Tcore is the engine temperature; d(x,y) is the distance from position (x,y) to the center of the engine; σ is the temperature attenuation coefficient, usually set to σ=0.2L; L is the vehicle length; Tenv is the ambient temperature;

[0151] The above formula simulates the temperature distribution spreading outward from the engine through exponential decay, while taking into account the influence of ambient temperature. This model can generate infrared thermal imaging data of the vehicle and provide infrared spectral data for the multimodal dataset.

[0152] S2. Use terrain elevation data to construct simulated terrain; the specific method is as follows:

[0153] S201. Initialize parameters, including terrain roughness parameter H and number of iterations n;

[0154] In a specific embodiment, the terrain roughness parameter H is set to 0.7. This parameter is used to control the roughness of the terrain. The smaller the value, the rougher the terrain; the larger the value, the smoother the terrain. By setting H to 0.7, an elevation map with natural terrain characteristics can be generated, simulating the unevenness and complexity of the actual terrain.

[0155] In a specific embodiment, the number of iterations n is set to 8. This parameter determines the level of detail of the terrain elevation generated. The greater the number of iterations, the richer the generated terrain details. By setting n to 8, sufficiently detailed terrain elevation data can be generated while ensuring computational efficiency.

[0156] S202. Generate multi-scale noise data using Perlin noise function; for each iteration k , calculate the noise function noise(2 k x ,2 k y ), and perform weighted superposition according to the formula;

[0157] Perlin noise is a gradient noise with the characteristics of continuity and directionality. It can generate a more natural and smooth noise pattern and is suitable for generating natural terrain. By using the Perlin noise function, the block effect can be effectively avoided and more realistic terrain elevation data can be generated.

[0158] S203. Calculate terrain elevation: For each location ( x , y ), calculate the elevation value according to the following formula H ( x , y ):

[0159] ;

[0160] Where: H ( x , y ) means at position ( x , y )’s terrain elevation value; n Indicates the number of iterations, that is, the number of layers of noise function calculation; k Indicates the index of the current iteration, from 0 to n ; 2 -kH represents the weight of each iteration;

[0161] S204. Use the generated terrain elevation data to construct a simulated terrain.

[0162] S3 target motion data set construction: the vehicle model generated in step S1 runs at various speeds on the simulated terrain constructed in step S2, generating a sequence of images containing various motion speeds, random lens shake and different lighting conditions, and constructing a target motion data set;

[0163] S301. Define the vehicle motion trajectory, simulate the vehicle's posture changes on complex terrain, and build a vehicle motion trajectory model; specifically including:

[0164] S3011. Define vehicle trajectory: The vehicle moves at a speed of v (Unit: pixels / second) Moves along the terrain surface, and its position changes with time t The change is expressed as:

[0165] x vehicle( t )= x 0+ v · t

[0166] y vehicle( t )= y 0

[0167] in: x 0. y 0 means the vehicle is x Axis and y The initial position of the axis; y 0 is fixed to the terrain center line; v ∈{1,2,4,8} pixels / second, which is the preset speed value;

[0168] The motion speed is set to 1 pixel / s, 2 pixels / s, 4 pixels / s, 8 pixels / s, etc.;

[0169] S3012. Based on terrain elevation H ( x , y ) Calculate the vehicle's roll angle i roll and pitch angle i pitch , simulating the vehicle's posture changes on complex terrain; roll angle i roll and pitch angle i pitch It is expressed as follows:

[0170] ;

[0171] Where: Δ x , Δ y A small displacement (such as 0.1 pixel) used to calculate terrain gradient.

[0172] S302 simulates the translational shaking and rotational shaking of the UAV during flight, and establishes a translational shaking model and a rotational shaking model;

[0173] The translational shaking model is as follows:

[0174] Camera translation shake offset Δ x cam , Δ y cam Obey a first-order autoregressive process:

[0175] ;

[0176] Where: α is the attenuation factor (α=0.9), which controls the persistence of the shaking; s x 、 s y Represents pixels ( s x = s y =0.5), controlling the translation amplitude; 、 is random noise;

[0177] The rotational shaking model is as follows:

[0178] Camera rotation angle i cam follows an autoregressive process:

[0179] ;

[0180] in: α is the attenuation factor; represents random noise, s θ Indicates radians ( s θ =0.1).

[0181] S303. The illumination angle is incremented by 5 minutes, simulating the solar azimuth and solar altitude at different time periods to establish an illumination angle progression model;

[0182] Solar azimuth ϕ light The calculation formula is as follows:

[0183] ;

[0184] Among them, g represents the g Group data; Indicates the initial east direction, =0°; =1.25°, corresponding to the angle of the Earth's rotation in 5 minutes (15° per hour);

[0185] Sun altitude angle i light The calculation formula is as follows:

[0186] ;

[0187] Among them, m represents the m Group data; t total Indicates the simulation time in minutes; i max is the maximum altitude angle (at noon), i max =60°; T Represents a 12-hour sunshine cycle, T =720 minutes.

[0188] S304. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate 500 sets of visible light and infrared images, each consisting of 100 frames of image sequences, to form a multispectral image dataset. The specific operations are as follows:

[0189] S3041. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate a visible light image and an infrared image for each frame; render the generated visible light image and infrared image; the rendering method specifically includes:

[0190] For each set of data g and each frame k (0≤ k <100) Perform the following operations:

[0191] Calculation time t = k ·Δ t Where, Δ t Describe the frame interval, Δ t =0.1 seconds (10fps);

[0192] according to x vehicle( t ) and terrain elevation H ( x , y ), adjust the vehicle posture;

[0193] Overlay camera translation shake offset (Δ x cam ( k ),Δ y cam ( k )) and camera rotation angle i cam ( k );

[0194] Rendering a visible light image and an infrared image based on the multi-spectral vehicle model data of step S102 in combination with the solar azimuth angle and the solar altitude angle;

[0195] S3042. Apply affine transformation to the rendered image to simulate camera shake effect:

[0196] ;

[0197] Where: i cam is the rotation angle caused by camera shake; Δ x cam and Δ y camis the translation offset caused by camera shake; ( x , y ) are the pixel coordinates in the original image; ( x ′, y ′) is the new coordinate after affine transformation;

[0198] S3043. After rendering, 500 groups of visible light images and infrared images are generated, each group including 100 frames of sequence images, to form a multi-spectral image dataset;

[0199] The resolution of visible light images and infrared images is set according to requirements.

[0200] In this step, metadata includes vehicle trajectories, camera shake parameters, illumination angles, and speed labels. This model generates a multispectral image dataset containing complex motion trajectories, dynamic illumination, and random camera shake, simulating the observation conditions of drones in real flight scenarios and providing robust training data for deep learning models.

[0201] S4. Extract the target motion trajectory of each set of sequence images in step S3 using the motion trajectory extraction method of multi-interval optical flow fusion, and pair it with the image data in step S3 to form a training data set; specifically, the following sub-steps are included:

[0202] S401. Input data preprocessing: Input the 100-frame image sequence (resolution 1920×1080) generated in step S3, and fuse the visible light (RGB) image and infrared (IR) image to generate a multi-spectral grayscale image. This includes:

[0203] S4011. Convert the visible light (RGB) image to a single-channel grayscale image; the conversion formula is:

[0204] Gray rgb =0.299×R+0.587×G+0.114×B;

[0205] Where R, G, and B represent the pixel values ​​of the red, green, and blue channels respectively;

[0206] S4012. Normalize the infrared (IR) image to adjust the pixel value range to 0 to 255. The normalization formula is:

[0207] ;

[0208] Of which: IR min Represents the normalized infrared image; IR min and IR max are the minimum pixel value and maximum pixel value of the current infrared image respectively;

[0209] S4013. Perform weighted fusion of the grayscale image and the normalized infrared image to generate a multi-spectral grayscale image. The fusion formula is:

[0210] Gray fused =α×Gray rgb +(1-α)×IR norm ;

[0211] Among them, Gray fused represents the fused multi-spectral grayscale image, α represents the fusion weight coefficient, α is adjustable in the range [0.4, 0.7], and α=0.6 is preferred in the specific embodiment;

[0212] S4014. Pyramid construction: using the fused multi-spectral grayscale image Gray fused Based on the image, a 4-layer image pyramid is created (scaling factor 0.5). Pyramid downsampling is used to reduce the computational workload and enhance the robustness of optical flow calculation, reducing the top-level resolution to 240×135.

[0213] S402. Using the data pre-processed in step S401, multi-interval optical flow calculation is performed; the multi-interval optical flow calculation includes short-interval, medium-interval, and long-interval optical flow calculations; specifically, the following steps are performed:

[0214] Interval set definition: Define the interval set Δset, including short intervals, medium intervals and long intervals:

[0215] Short interval (Δ≤5): intensive frame-by-frame calculation to capture high-speed targets (such as vehicles); the specific calculation formula for the short-interval optical flow field is as follows:

[0216] ;

[0217] in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, window=5×5 means the window size is 5×5;

[0218] Intermediate interval (5<Δ≤20): sparse calculation is performed every other frame to balance speed and noise. The specific calculation formula of the intermediate interval optical flow field is as follows:

[0219] ;

[0220] in, I t and I t+Δ Respectively represent t Frame and t+Δ frame image; poly_n=7 represents the polynomial kernel size. The polynomial kernel is used to fit the change of image grayscale value to estimate the movement of pixel points. 7 is the order of the polynomial kernel, that is, the highest degree of the polynomial.

[0221] Long interval (Δ>20): Global calculation across frames to enhance the signal-to-noise ratio of low-speed targets; the specific calculation formula for the long interval optical flow field is as follows:

[0222] ;

[0223] in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, edge_th=0.1 is the edge threshold, which is used to control the accuracy of optical flow calculation.

[0224] The optical flow calculation method in this step briefly explains its principle: Single-interval optical flow methods (such as Horn-Schunck) are only suitable for small displacements and cannot balance speed and range. This method uses a three-stage parallel calculation, combining short-interval, medium-interval, and long-interval optical flow, to cover targets across a wide range of speeds. This method clearly reveals motion changes compared to using continuous frames. Short-interval optical flow is used to capture high-speed targets, medium-interval optical flow balances speed and noise, and long-interval optical flow enhances the signal-to-noise ratio of low-speed targets.

[0225] S403. Dynamic weight allocation: Adaptively adjust the fusion weight based on the target area speed to optimize the tracking effect of targets with different speeds. The fusion formula is as follows:

[0226] ;

[0227] in, is the fused optical flow field; α 、 β 、 c represents weight;

[0228] For high-speed targets (speed v >50 pixels / frame): α =0.3, β =0.5, c =0.2; focuses on the middle interval optical flow to prevent blur;

[0229] For low-speed targets (speed v ≤50 pixels / frame): α =0.6, β =0.3, c =0.1; focuses on short-interval optical flow to preserve details;

[0230] S404. Trajectory extraction: Based on the fused optical flow field , update the target position corresponding to each frame image P t , calculate the target motion trajectory frame by frame;

[0231] ;

[0232] S405 trajectory pairing: the target motion trajectory obtained in step S404 is paired with the target motion data set image in step S3, and the corresponding motion trajectory data is found for each frame image to form a training data set;

[0233] In this step, the training dataset created is used to train a machine learning model to predict or identify target motion in image sequences. This step prepares the training data, ensuring that each frame has corresponding trajectory information so that the model can learn the relationship between image content and motion. Through these steps, the target's motion trajectory can be effectively extracted and paired with the image data, providing high-quality training data for the deep learning model and improving the model's ability to detect dynamic targets.

[0234] S5. Build a deep learning network based on the dual-stream spatiotemporal attention mechanism and train the deep learning network: Input the training dataset (target motion trajectory and target motion image) obtained by pairing in step S4, use the dynamic course learning training method to train the network, and store the weight file after training. This includes the following sub-steps:

[0235] S501. Multi-spectral input data stitching: Perform affine transformation registration on the RGB (3-channel) and IR (1-channel) images at the same time. Concatenate the registered IR channel with the RGB channel along the channel dimension to form a 4-channel input (R / G / B / IR) with a size of 256×256×4.

[0236] S502. Construct a deep learning network based on the dual-stream spatiotemporal attention mechanism;

[0237] The deep learning network based on the dual-stream spatiotemporal attention mechanism includes a multi-spectral feature extraction branch, a motion trajectory embedding branch, and a cross-modal fusion module connected in sequence;

[0238] The network structure of the multi-spectral feature extraction branch adopts the improved ResNet-50 skeleton network ( Figure 3 ), adding a spectral attention module (Spectral Attention Module) to enhance the representation ability of multi-spectral features ( Figure 4The first convolution kernel is adjusted to 7×7×4 to accommodate four-channel input. The input is a RGB-IR four-channel spliced ​​image (256×256×4). The spectral attention module outputs the weighted feature map SA(F). The feature representation capability is enhanced using the following calculation formula:

[0239] SA(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) F;

[0240] Where: SA(F) represents the output of the spectral attention module; F represents the feature map; σ is the sigmoid activation function; MLP represents the multi-layer perceptron; AvgPool(F) represents the global average pooling of the feature map F; MaxPool(F) represents the global maximum pooling of the feature map F; Represents element-wise multiplication;

[0241] The advantages of the multi-spectral feature extraction branch architecture of the present invention compared with the traditional ResNet-50 architecture are shown in Table 1.

[0242] Table 1 Architecture diagram of the multi-spectral feature extraction branch Advantages

[0243]

[0244] The motion trajectory embedding branch uses a 3D convolutional temporal encoder, which contains four (3×3×3) convolutional layers to capture spatiotemporal features. The spatiotemporal position encoding is achieved by the following formula:

[0245] ;

[0246] Where: pos is the position index of the element in the sequence; i is the dimension index, starting from 0; d model is the dimension of the position encoding vector;

[0247] The cross-modal fusion module uses a gated cross-attention mechanism to dynamically fuse spectral and motion features. The calculation process is as follows:

[0248] Q=LN(F spectral )

[0249] K=V=LN(F motion )

[0250] GatedAttn=Softmax(QK T / )⊙Sigmoid(Conv(Q))

[0251] F fusion =GatedAttn V;

[0252] Where: Q, K, V represent query, key, and value respectively; LN represents layer normalization; F spectral represents the feature map obtained from the multi-spectral feature extraction branch; F motion represents the feature map obtained from the motion trajectory embedding branch; QK T Indicates Q and K T The dot product of represents the scaling factor of the dimension; ⊙ is the Hadamard product; F fusion Represents the fused feature map, the output obtained by the cross-modal fusion module; Represents a tensor product.

[0253] The technical advantages of the deep learning network based on the dual-stream spatiotemporal attention mechanism in this invention compared with the existing traditional network are shown in Table 2.

[0254] Table 2 Advantages of deep learning networks based on dual-stream spatiotemporal attention mechanism

[0255]

[0256] S503. Network training is performed using a dynamic curriculum learning training method, which includes three stages: basic training, multimodal fusion, and robustness enhancement. During the training process, a multi-task joint training strategy is used to optimize network performance. After training is completed, the optimal weight file of the network is exported and the network structure and parameter configuration are stored.

[0257] The basic training phase uses lower resolution (128×128) RGB images for preliminary training to learn basic feature representations. This phase uses a cosine decay learning rate schedule with an initial learning rate of 1e-3.

[0258] Multimodal fusion stage: Increase the input image resolution to 256×256, gradually introduce infrared images and optical flow data, so that the network can learn more complex multimodal features; in this stage, reset the learning rate to 5e-4 and adjust the data mixing ratio (0.2→1.0);

[0259] The robustness enhancement stage enhances the robustness of the network by adding random motion blur (kernel size 5-15) and simulating extreme illumination changes (±30% brightness perturbation); at the same time, the exponential moving average (EMA) strategy is enabled to stabilize the training process with a decay rate of 0.999.

[0260] The multi-task joint training strategy adopts a composite loss function that combines target detection loss, optical flow regression loss, and motion consistency loss to optimize network performance;

[0261] Define the composite loss function: L = l 1 L det + l 2 L flow + l 3 L reg ;

[0262] Where, L det represents the target detection loss; L flow represents the optical flow regression loss; L reg Indicates motion consistency loss; l 1 、l 2 、l 3 represents the hyperparameter, l 1=1.0, l 2=0.5, l 3=0.2;

[0263] Object detection loss L det Using the improved Focal Loss loss function, parameter adjustment factor c =3.0, weight α =0.8;

[0264] Optical flow regression loss L flow Using Charbonnier loss:

[0265]

[0266] Where, x and y are the predicted value and the true value, and the parameter =1 e -3;

[0267] Motion consistency loss L reg As a trajectory smoothness constraint, the calculation formula is:

[0268] ;

[0269] Where, p t It's in time t The predicted position of Δ p t It's in time t The true displacement of Represents the Euclidean norm (L2 norm), which is used to calculate the distance between vectors;

[0270] After training is completed, the optimal weight file is exported to store the network structure and parameter configuration.

[0271] Through this phased training strategy, the network can gradually learn and adapt to complex multimodal data at different stages, ultimately achieving efficient detection of dynamic targets.

[0272] S6. In the actual flight mission, call the weight file to identify the moving target; the specific steps include:

[0273] S601. During the flight mission, use both a visible light camera and an infrared camera to capture visible light and infrared images of the target. Use the SURF (Speeded Up Robust Features) feature detector to identify key feature points in the images. Perform affine transformation registration based on the detected feature points and calculate the affine transformation matrix T:

[0274] T=estimateAffine2D(rgb kpts ,ir kpts );

[0275] ir aligned =warpAffine(ir img ,T,(width,height));

[0276] Where: rgb kpts and ir kpts Represent the key feature points of the visible light image and infrared image respectively; the input of the estimateAffine2D algorithm is two sets of corresponding feature points, and the output is the 2D affine transformation matrix T; warpAffine represents the image transformation function; (width, height) represents the size of the transformed image;

[0277] S602. Perform motion compensation based on the inertial measurement unit (IMU) data and correct the aligned images by applying a homography warp. The formula is as follows:

[0278] compensated img =homographyWarp(raw img ,H inv );

[0279] In order to further smooth the image, Kalman filtering is used, where the process noise covariance is set to Q = 1e-5 and the measurement noise covariance is R = 0.01;

[0280] The de-shaking image will be sent to the online inference optimization stage for subsequent processing;

[0281] S603. Enable mixed-precision computing (using FP16 and INT8 formats) when deploying on embedded boards to improve computing efficiency. Dynamically adjust the resolution of the de-jittered image based on the flight altitude to obtain an optimized image. The resolution calculation formula is:

[0282] Resolution=[1080×altitude / 100];

[0283] Where: Resolution represents the calculated resolution; altitude represents the flight altitude in meters; 1080 represents the base resolution width, and 100 is the scale factor. This step allows the system to adaptively adjust the processing resolution at different flight altitudes to balance the computational load and accuracy requirements.

[0284] The optimized image data is fed into the motion trajectory prediction module for further target tracking and analysis. Through this adaptive computational scheduling approach, the system is able to maintain efficient online inference performance under varying flight conditions while ensuring accurate target detection and tracking.

[0285] The UAV is embedded in a motion trajectory prediction module. It uses the Long Short-Term Memory (LSTM) prediction unit integrated within the module to predict its position for the next three frames. The LSTM prediction unit contains 64 hidden layer units, which are used to process time series data and predict future motion trajectories. The prediction equation is as follows:

[0286] h t =σ(W xh x t +W hh h t-1 +b h );

[0287] p t+1 =W hp h t +b p ;

[0288] Where: h t and h t−1 Represent the hidden states of the current time step and the previous time step respectively; σ is the activation function; W xh 、W hh 、W hp represents the weight matrix; x t is the current input; b h and b p are the bias terms added to the hidden state and added to the output respectively;

[0289] S604. Result post-processing and fusion: The weighted box fusion (WBF) algorithm is used to aggregate detection results of different scales to improve detection accuracy and robustness. The fusion formula is as follows:

[0290] fused box =∑(w i box i ) / ∑w i

[0291] w i =confidence i ×IoU i

[0292] In the formula: fused box represents the bounding box after fusion, w i is the weight of the detection box, box i Represents the i-th detection box, confidence i Represents confidence, IoU i represents the intersection and union ratio;

[0293] Compare the intersection-over-union (IoU) of the visible light image and the infrared image. If the IoU is less than 0.3, call the reject_low_confidence() function to reject the low-confidence detection result.

[0294] Generates visualization results with trajectory prediction (frame rate 30fps) so that users can monitor and analyze the target's motion trajectory in real time.

[0295] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0296] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for detecting moving targets in dynamic flight of an unmanned aerial vehicle, characterized by: The specific steps include: S1. Generate vehicle models of different sizes and types through a parametric generation method, build a parametric model library containing typical vehicles, generate multi-spectral vehicle model data, and construct a vehicle dataset; S2. Constructing simulated terrain using terrain elevation data; S3 target motion data set construction: the vehicle model generated in step S1 runs at various speeds on the simulated terrain constructed in step S2, generating a sequence of images containing various motion speeds, random lens shakes and different lighting conditions, and constructing a target motion data set; S4. Extract the target motion trajectory of each set of sequence images in step S3 using the motion trajectory extraction method of multi-interval optical flow fusion, and pair it with the image data in step S3 to form a training data set; S5. Construct a deep learning network based on the dual-stream spatiotemporal attention mechanism; Input the training data set obtained by pairing in step S4, use the dynamic curriculum learning training method to train the network, and store the weight file after training; S6. In actual flight missions, call the weight file to identify moving targets.

2. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The step S1 specifically includes the following sub-steps: S101. Define the vehicle length, width and height parameters as: ； Among them, the vehicle length L conforms to the uniform distribution with parameters lmin~lmax, lmin is the minimum length of a typical vehicle, and lmax is the maximum length of a typical vehicle; U is a uniform distribution, N is a Gaussian distribution, ΔW and ΔH To simulate the dimensional deviation of the actual vehicle; S102. Generate multi-spectral vehicle model data for each generated vehicle model, including visible light image data and infrared image data; construct a vehicle dataset using the generated multi-spectral vehicle model data; The method for generating visible light image data includes: simulating the optical reflection characteristics of the vehicle surface according to an optical reflection amount formula to generate visible light image data; the optical reflection amount formula is expressed as follows: ； in: represents the reflection function; θ i represents the angle of incidence; θ o represents the exit angle; θ h It represents the angle between the half angle and the surface normal; α represents the surface roughness parameter; F 0 represents the value of Fresnel reflectivity when the incident angle is 0; θ d It represents the angle of incidence θ i and the exit angle θ o The difference between The method for generating infrared image data comprises: establishing a temperature gradient model between the engine area and the non-engine area, generating a temperature distribution on the vehicle surface according to the temperature gradient model; and generating infrared image data according to the temperature distribution. The temperature distribution on the vehicle surface is expressed as follows: ； Where: T(x,y) is the temperature at position (x,y); Tcore is the engine temperature; d(x,y) is the distance from position (x,y) to the center of the engine, σ is the temperature attenuation coefficient; L is the vehicle length; Tenv is the ambient temperature.

3. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The step S2 specifically includes the following sub-steps: S201. Initialize parameters, including terrain roughness parameter H and number of iterations n; S202. Generate multi-scale noise data using Perlin noise function; for each iteration k , calculate the noise function noise(2 k x ,2 k y ), and perform weighted superposition according to the formula; S203. Calculate terrain elevation: For each location ( x , y ), calculate the elevation value according to the following formula H ( x , y ): ； Where: H ( x , y ) means at position ( x , y )’s terrain elevation value; n Indicates the number of iterations, that is, the number of layers of noise function calculation; k Indicates the index of the current iteration; 2 -kH represents the weight of each iteration; S204. Use the generated terrain elevation data to construct a simulated terrain.

4. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The step S3 specifically includes the following sub-steps: S301 defines the vehicle motion trajectory, simulates the vehicle's posture changes on complex terrain, and constructs a vehicle motion trajectory model; S302 simulates the translational shaking and rotational shaking of the UAV during flight, and establishes a translational shaking model and a rotational shaking model; S303. The illumination angle is incremented by 5 minutes, simulating the solar azimuth and solar altitude at different time periods to establish an illumination angle progression model; S304. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate 500 sets of visible light images and infrared images, each set including 100 frames of sequential images, to form a multi-spectral image dataset.

5. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 4, wherein: The step S301 specifically includes: S3011. Define vehicle trajectory: The vehicle moves at a speed of v Movement along the terrain surface, position over time t The change is expressed as: x vehicle( t )= x 0+ v · t y vehicle( t )= y 0; in: x 0. y 0 means the vehicle is x Axis and y The initial position of the axis; y 0 is fixed to the terrain center line; v The unit is pixels / second; S3012. Based on terrain elevation H ( x , y ) Calculate the vehicle's roll and pitch angles to simulate the vehicle's posture changes on complex terrain; The step S304 specifically includes: S3041. Combine the vehicle motion trajectory model, the UAV translational and rotational sway models, and the illumination angle progression model to generate a visible light image and an infrared image for each frame; render the generated visible light image and infrared image; the rendering method specifically includes: For each set of data g and each frame k Perform the following operations: Calculation time t = k ·Δ t Where, Δ t Express the frame interval; according to x vehicle( t ) and terrain elevation H ( x , y ), adjust the vehicle posture; Overlay camera translation shake offset (Δ x cam ( k ),Δ y cam ( k )) and camera rotation angle θ cam ( k ); Render visible light and infrared images; S3042. Apply affine transformation to the rendered image to simulate camera shake effect: ； Where: θ cam is the rotation angle caused by camera shake; Δ x cam and Δ y cam is the translation offset caused by camera shake; S3043. After rendering, 500 groups of visible light images and infrared images are generated, each group including 100 frames of sequential images, to form a multi-spectral image dataset.

6. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The step S4 specifically includes the following sub-steps: S401. Input data preprocessing: Input the 100-frame sequence image generated in step S3, and fuse the visible light image and infrared image to generate a multi-spectral grayscale image; S402. Using the data pre-processed in step S401, multi-interval optical flow calculation is performed; multi-interval optical flow calculation includes short interval, medium interval and long interval optical flow calculation; The specific calculation formula of the short-interval optical flow field is as follows: ； in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, window represents the window size; The specific calculation formula of the intermediate optical flow field is as follows: ； in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image; poly_n=7 represents the polynomial kernel size; ； in, I t and I t+Δ Respectively represent t Frame and t +Δ frame image, edge_th=0.1 is the edge threshold; S403. Dynamic weight allocation: Adaptively adjust the fusion weight based on the target area speed to optimize the tracking effect of targets with different speeds. The fusion formula is as follows: ； in, is the fused optical flow field; α 、 β 、 γ represents weight; S404. Trajectory extraction: Based on the fused optical flow field , update the target position corresponding to each frame image P t , calculate the target motion trajectory frame by frame; S405. Trajectory pairing: Pair the target motion trajectory obtained in step S404 with the images in the target motion dataset in step S3, find the corresponding motion trajectory data for each frame of image, and form a training dataset.

7. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 6, wherein: The step S401 specifically includes: S4011. Convert the visible light image into a single-channel grayscale image; the conversion formula is: Gray rgb =0.299×R+0.587×G+0.114×B; Where R, G, and B represent the pixel values ​​of the red, green, and blue channels respectively; S4012. Normalize the infrared image and adjust the pixel value range to 0 to 255. The normalization formula is: ； Of which: IR min and IR max are the minimum pixel value and maximum pixel value of the current infrared image respectively; S4013. Perform weighted fusion of the grayscale image and the normalized infrared image to generate a multi-spectral grayscale image. The fusion formula is: Gray fused =α×Gray rgb +(1-α)×IR norm ; Among them, Gray fused represents the fused multi-spectral grayscale image, and α represents the fusion weight coefficient; S4014. Pyramid construction: using the fused multi-spectral grayscale image Gray fused Based on the image pyramid, a 4-layer image pyramid is created, and the top layer resolution is reduced to 240×135 by pyramid downsampling.

8. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The deep learning network based on the dual-stream spatiotemporal attention mechanism in step S5 includes a multi-spectral feature extraction branch, a motion trajectory embedding branch and a cross-modal fusion module; The network structure of the multi-spectral feature extraction branch adopts an improved ResNet-50 backbone network and adds a spectral attention module to enhance the representation capability of multi-spectral features. The first-layer convolution kernel is adjusted to 7×7×4. The input is an RGB-IR four-channel spliced ​​image. The spectral attention module outputs a weighted feature map SA(F). The feature representation capability is enhanced using the following calculation formula: SA(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) F; Where: SA(F) represents the output of the spectral attention module; F represents the feature map; σ is the sigmoid activation function; MLP represents the multi-layer perceptron; AvgPool(F) represents the global average pooling of the feature map F; MaxPool(F) represents the global maximum pooling of the feature map F; Represents element-wise multiplication; The motion trajectory embedding branch uses a 3D convolutional temporal encoder, which consists of four 3×3×3 convolutional layers. The spatiotemporal position encoding is implemented by the following formula: ； Where: pos is the position index of the element in the sequence; i is the dimension index, starting from 0; d model is the dimension of the position encoding vector; The cross-modal fusion module uses a gated cross-attention mechanism to dynamically fuse spectral and motion features. The calculation process is as follows: Q=LN(F spectral ) K=V=LN(F motion ) GatedAttn=Softmax(QK T / )⊙Sigmoid(Conv(Q)) F fusion =GatedAttn V; Where: Q, K, V represent query, key, and value respectively; LN represents layer normalization; F spectral represents the feature map obtained from the multi-spectral feature extraction branch; F motion represents the feature map obtained from the motion trajectory embedding branch; QK T Indicates Q and K T The dot product of represents the scaling factor of the dimension; ⊙ is the Hadamard product; F fusion Represents the fused feature map, the output obtained by the cross-modal fusion module; Represents a tensor product.

9. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 8, wherein: The step S5 specifically includes: S501. Multi-spectral input data stitching: Perform affine transformation registration on the visible light image and infrared image at the same time. Then stitch the registered infrared channel and visible light channel along the channel dimension to form a 4-channel input with a size of 256×256×4. S502. Construct the deep learning network based on the dual-stream spatiotemporal attention mechanism; S503. Network training is performed using a dynamic curriculum learning training method, including a basic training phase, a multimodal fusion phase, and a robustness enhancement phase. During the training process, a multi-task joint training strategy is used to optimize network performance. After training is completed, the optimal weight file of the network is exported and the network structure and parameter configuration are stored. In the basic training phase, 128×128 resolution visible light images are used for preliminary training to learn basic feature representations; a cosine decay learning rate schedule is used, with an initial learning rate of 1e-3; In the multimodal fusion stage, the input image resolution is increased to 256×256, and infrared images and optical flow data are gradually introduced to enable the network to learn more complex multimodal features; the learning rate is reset to 5e-4; The robustness enhancement stage enhances the robustness of the network by adding random motion blur and simulating extreme lighting changes; The multi-task joint training strategy uses a composite loss function to optimize network performance; the composite loss function is defined as: L = λ 1 L det + λ 2 L flow + λ 3 L reg ; Where, L det represents the target detection loss; L flow represents the optical flow regression loss; L reg Indicates motion consistency loss; λ 1 ,λ 2 、 λ 3 represents hyperparameters; Object detection loss L det The improved Focal Loss loss function is used to adjust the parameter factor γ =3.0, weight α =0.8; Optical flow regression loss L flow Using Charbonnier loss: ； Where, x and y are the predicted value and the true value, and the parameter =1 e −3; Motion consistency loss L reg As a trajectory smoothness constraint, the calculation formula is: ； Where, p t It's in time t The predicted position of Δ p t It's in time t The true displacement of Represents the Euclidean norm, which is used to calculate the distance between vectors; After training is completed, the optimal weight file is exported to store the network structure and parameter configuration.

10. The method for detecting a moving target in a dynamic flight of an unmanned aerial vehicle according to claim 1, wherein: The step S6 specifically includes: S601. During the flight mission, use a visible light camera and an infrared camera to simultaneously capture visible light and infrared images of the target. Use a SURF feature detector to identify key feature points in the image. Perform affine transformation registration based on the detected feature points and calculate the affine transformation matrix T: T=estimateAffine2D(rgb kpts ,ir kpts ); ir aligned =warpAffine(ir img ,T,(width,height)); Where: rgb kpts and ir kpts Represent the key feature points of the visible light image and infrared image respectively; the input of the estimateAffine2D algorithm is two sets of corresponding feature points, and the output is the 2D affine transformation matrix T; warpAffine represents the image transformation function; (width, height) represents the size of the transformed image; S602. Motion compensation is performed based on the inertial measurement unit data, and the aligned images are corrected by applying a homography transformation; Kalman filtering is used to process the de-jittered images and send them to the online inference optimization stage; S603. Enable mixed precision computing when deploying embedded boards; dynamically adjust the resolution of the de-jittered image based on the flight altitude to obtain an optimized image; the optimized image data is fed into the motion trajectory prediction module, which predicts the position of the next three frames based on the long short-term memory network prediction unit integrated within the module; S604. Result post-processing and fusion: A weighted box fusion algorithm is used to aggregate detection results of different scales to improve detection accuracy and robustness. The fusion formula is as follows: fused box =∑(w i ·box i ) / ∑w i w i =confidence i ×IoU i In the formula: fused box represents the bounding box after fusion, w i is the weight of the detection box, box i Represents the i-th detection box, confidence i Represents confidence, IoU i represents the intersection and union ratio; Compare the intersection-over-union (IoU) of the visible light image and the infrared image. If the IoU is less than 0.3, call the reject_low_confidence() function to reject the low-confidence detection result; generate a visualization result with trajectory prediction.

Images