A multi-target detection and tracking system for unmanned aerial vehicles

Abstract

The present invention relates to the technical field of target detection and tracking, and discloses an unmanned aerial vehicle multi-target detection and tracking system, including a data acquisition and preprocessing module, a target detection module, a motion trajectory dynamic modeling module and a target classification module: the data acquisition and preprocessing module is used to acquire optical image data, three-dimensional point cloud data and motion data of the target, and perform preprocessing to obtain first feature data, second feature data and third feature data respectively; the target detection module is used to extract image texture features and spatial geometric features, and perform fusion matching; the motion trajectory dynamic modeling module is used to generate a motion vector field according to the first feature data, correct it through the third feature data, and perform cluster analysis to obtain dynamic motion features; the target classification module is used to aggregate image texture features and dynamic motion features to generate target classification results. The present invention can adapt to complex dynamic scenes and occluded environments, and has wide applicability.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection and tracking, and in particular relates to an unmanned aerial vehicle-borne multi-target detection and tracking system. Background Art

[0002] With the rapid development of drone technology, its application in military, security, logistics and other fields is becoming more and more extensive. In complex dynamic environments, the accurate tracking of targets by drones is a key capability to achieve task automation. Current target detection and tracking mainly rely on computer vision technology, which uses image feature extraction and deep learning algorithms to achieve target recognition and motion trajectory tracking. However, there are still obvious deficiencies in target tracking tasks in actual scenarios. On the one hand, when the motion trajectory of the target changes dramatically (such as rapid acceleration and sharp turns), traditional methods are limited by fixed motion models and it is difficult to quickly adapt to the nonlinear motion mode of the target, resulting in slow response of the tracking system or loss of the target. On the other hand, existing technologies usually rely on information from a single modality (such as image features). When the target is occluded or in complex scenes, there is a lack of deep modeling and fusion of motion information, and the tracking accuracy is prone to significant decline. Summary of the invention

[0003] The present invention provides an unmanned aerial vehicle (UAV)-mounted multi-target detection and tracking system, which solves the technical problems of slow response speed, insufficient real-time performance and low tracking accuracy in related technologies under scenes of rapid target change and dynamic occlusion.

[0004] The present invention provides an unmanned aerial vehicle multi-target detection and tracking system, including a data acquisition and preprocessing module, a target detection module, a motion trajectory dynamic modeling module and a target classification module:

[0005] A data acquisition and preprocessing module, used to acquire optical image data, three-dimensional point cloud data and motion data of the target, and preprocess the above data to obtain first feature data, second feature data and third feature data respectively;

[0006] Among them, motion data includes the position, speed and direction of the target;

[0007] A target detection module is used to extract image texture features according to the first feature data and generate a target candidate region according to a self-attention mechanism, extract spatial geometric features of the target in the target candidate region according to the second feature data, fuse and match the image texture features with the spatial geometric features, and obtain a target region containing image texture features and spatial geometric features;

[0008] A motion trajectory dynamic modeling module, used to generate a motion vector field according to the first feature data, correct the motion vector field according to the third feature data, and perform cluster analysis on the corrected motion vector field to obtain dynamic motion features with cluster classification results;

[0009] The target classification module is used to aggregate image texture features and dynamic motion features to generate target classification results.

[0010] Furthermore, the specific steps of the pretreatment include:

[0011] Step S201, using the image frame of the optical image data Perform denoising, where represents the Gaussian function value, and Respectively represent the standard deviation in the horizontal and vertical directions, exp represents the exponential function, and represent the velocity attenuation coefficients in the horizontal and vertical directions, respectively; x and y represent the coordinates of the pixels of the optical image data, respectively; represents the smoothing coefficient, and Respectively represent the gradients of the image frame in the horizontal and vertical directions;

[0012] Step S202, calculating the normal vector of each point of the three-dimensional point cloud data by constructing a plane equation, and splicing it behind the three-dimensional point cloud data;

[0013] Step S203: when it is determined that the weighted neighborhood density of a data point of the three-dimensional point cloud data is less than a first preset density, the data point is regarded as a noise point and removed. The calculation formula of the weighted neighborhood density of the data point is: ,in, Indicate point The weighted neighborhood density of Data points representing 3D point cloud data, Indicate point Neighborhood points, Represents the set of neighbor points within a preset radius, represents the distance attenuation coefficient, represents the distance attenuation weight coefficient, Indicate point and neighbors The Euclidean distance of

[0014] Step S204, using a Kalman filter method to smooth the position and speed of the target in the motion data;

[0015] Step S205 , adding a unified timestamp to the processed optical image data, three-dimensional point cloud data and motion data, synchronizing the acquisition time of each data, and obtaining first feature data, second feature data and third feature data.

[0016] Further, an image feature extraction model is constructed according to the improved convolutional neural network, and the image feature extraction model is used to extract image texture features of the first feature data;

[0017] The image feature extraction model consists of an input layer, a multi-scale convolutional layer, a channel attention module, a multi-scale fusion layer, and an output layer;

[0018] The input layer is used to receive the first feature data;

[0019] The multi-scale convolution layer uses convolution kernels of different scales for feature extraction. Specifically, convolution kernels of different scales are used simultaneously in each convolution layer to extract and process the feature map, and the feature maps output by different convolution kernels are concatenated to obtain a multi-scale feature map.

[0020] The channel attention module is used to calculate the weight of each channel through global average pooling and perform weighted processing on the channels. The calculation formula of the channel attention module is: ,in, represents the feature map after channel attention weighting, represents a multi-scale feature map, and Represent the first weight matrix and the second weight matrix respectively, H and W represent the height and width of the multi-scale feature map respectively, h and w represent the index of the height and width respectively, c represents the index of the number of channels, sigmoid represents the sigmoid activation function, and ReLU represents the ReLU activation function;

[0021] The multi-scale fusion layer is used to concatenate the features extracted by the first half of the convolutional layer in the image feature extraction model with the features extracted by the second half of the convolutional layer in the channel dimension to obtain fused multi-scale features;

[0022] The output layer is used to reduce the dimension of multi-scale features through global average pooling and output image texture features.

[0023] Furthermore, the target candidate region is generated according to the self-attention mechanism, specifically including: obtaining the attention weight according to the image texture features and the formula of the self-attention mechanism, and when it is judged that the attention weight is greater than the preset attention weight threshold, it is used as the candidate target region. The formula of the self-attention mechanism is:

[0024] ;

[0025] in, represents the attention weight, express Activation function, , and Respectively represent query weight parameters, key weight parameters and value weight parameters, represents the image texture feature, T represents the transposition operation, and Respectively represent the horizontal distance and vertical distance between the mth position and the nth position, Represents the dimension of image texture features, Represents the minimum value between 0 and 1. express Activation function, represents the convolution operation, represents the local convolution kernel, The neighborhood area represents the texture features of the image.

[0026] Further, the spatial geometric features of the target in the target candidate area are extracted according to the second feature data, and the specific steps include:

[0027] Step S301, dividing the second feature data into voxel grids of a fixed size;

[0028] Step S302: for each non-empty voxel, input the point set into PointNet to extract the voxel local features of the voxel; specifically, use a multi-layer perceptron to perform feature mapping on each point in the voxel, and aggregate the features of all points in the voxel through a maximum pooling operation to obtain the voxel local features;

[0029] Step S303, performing a three-dimensional convolution operation on the voxel local features of each voxel to obtain spatial geometric features.

[0030] Furthermore, the image texture features are fused and matched with the spatial geometric features. The specific steps include:

[0031] Step S401, projecting the second feature data onto the target candidate area of ​​the first feature data to obtain a target projection area;

[0032] Step S402, sampling in the image texture feature according to the target projection area to obtain the point texture feature corresponding to each point in the second feature data;

[0033] Step S403, concatenating the spatial geometric features of each point in the second feature data with the corresponding point texture features, and finally obtaining a target area containing image texture features and spatial geometric features.

[0034] Furthermore, the motion vector of the pixel in the first feature data is calculated by the optical flow algorithm to generate an initial motion vector field, and the initial motion vector field is trajectory corrected by the Kalman filter algorithm using the third feature data to obtain a corrected motion vector field, wherein the calculation formula of the optical flow is: , by solving this formula, we can get the motion vector , , and They represent the horizontal gradient, vertical gradient and time gradient of the pixel respectively, and u and v represent the movement direction and speed of the pixel.

[0035] Furthermore, a motion feature vector is constructed for each motion vector in the corrected motion vector field, and the motion feature vector includes pixel coordinates, pixel motion direction and pixel speed; the corrected motion vector field is clustered using the K-means clustering algorithm to obtain K clusters, each cluster represents a motion pattern area, and the center point of each cluster represents the dynamic motion feature of the cluster, and the dynamic motion feature includes: average speed, average motion direction and motion pattern area range; wherein, the calculation formula of the average motion direction is: , represents the average motion direction of the kth cluster, arctan represents the inverse tangent function, represents the velocity of the i-th motion vector, represents the moving direction angle of the i-th motion vector, sin and cos represent the sine function and cosine function respectively; the motion pattern area range is determined by the bounding box of the cluster.

[0036] Furthermore, the image texture features and dynamic motion features of the target area are unified to the same feature dimension through a fully connected layer and spliced ​​to obtain fused features. The fused features are input into the first classifier to obtain target classification results. The target classification results include: target type and target dynamic behavior type.

[0037] The beneficial effects of the present invention are as follows: the present invention combines optical image data, three-dimensional point cloud data and motion data, extracts image texture features, spatial geometric features and dynamic motion features, and performs deep fusion matching; through the complementarity of multimodal features, it can significantly improve the detection and tracking accuracy of targets in complex scenes, and can accurately capture target information even when the target is partially blocked or the texture features are blurred;

[0038] The present invention generates an initial motion vector field through an optical flow algorithm and corrects the target motion trajectory in combination with a Kalman filter. This can effectively cope with scenarios where the target changes rapidly (such as acceleration and sharp turns) and improve the robustness of the tracking system. At the same time, the corrected motion vector field supports the analysis of dynamic behavior patterns and is suitable for various complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a module schematic diagram of a multi-target detection and tracking system carried by an unmanned aerial vehicle of the present invention;

[0040] Figure 2 It is a flow chart of the preprocessing of the present invention.

[0041] In the figure: data collection and preprocessing module 101, target detection module 102, motion trajectory dynamic modeling module 103 and target classification module 104. DETAILED DESCRIPTION

[0042] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0043] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0044] like Figure 1-Figure 2 As shown, a multi-target detection and tracking system carried by an unmanned aerial vehicle includes a data acquisition and preprocessing module 101, a target detection module 102, a motion trajectory dynamic modeling module 103 and a target classification module 104:

[0045] The data acquisition and preprocessing module 101 is used to acquire optical image data, three-dimensional point cloud data and motion data of the target, and preprocess the above data to obtain first feature data, second feature data and third feature data respectively;

[0046] Among them, motion data includes the position, speed and direction of the target;

[0047] The target detection module 102 is used to extract image texture features according to the first feature data and generate a target candidate region according to the self-attention mechanism, extract spatial geometric features of the target in the target candidate region according to the second feature data, fuse and match the image texture features with the spatial geometric features, and obtain a target region containing image texture features and spatial geometric features;

[0048] The motion trajectory dynamic modeling module 103 is used to generate a motion vector field according to the first feature data, correct the motion vector field according to the third feature data, and perform cluster analysis on the corrected motion vector field to obtain dynamic motion features with cluster classification results;

[0049] The target classification module 104 is used to aggregate image texture features and dynamic motion features to generate a target classification result.

[0050] In one embodiment of the present invention, an RGB camera is mounted on a UAV platform to collect optical image data of the target; a laser scanner is used to obtain three-dimensional point cloud data of the target, and the sampling frequency is synchronized with the frame rate of the camera to ensure that the point cloud and image collected at the same time can be aligned; a GPS positioning system is used to obtain the position, speed and direction of the target, and time synchronization is performed with the optical image data and three-dimensional point cloud data, the position is represented by coordinates, and the direction is represented by the sine and cosine values ​​of the direction angle.

[0051] In one embodiment of the present invention, the specific steps of the pre-processing include:

[0052] Step S201, using the image frame of the optical image data Perform denoising, where represents the Gaussian function value, and Respectively represent the standard deviation in the horizontal and vertical directions, exp represents the exponential function, and represent the velocity attenuation coefficients in the horizontal and vertical directions, respectively; x and y represent the coordinates of the pixels of the optical image data, respectively; represents the smoothing coefficient, and Respectively represent the gradient of the image frame in the horizontal and vertical directions, that is, the grayscale change rate in the horizontal and vertical directions;

[0053] Step S202, calculating the normal vector of each point of the three-dimensional point cloud data by constructing a plane equation, and splicing it behind the three-dimensional point cloud data;

[0054] Step S203: when it is determined that the weighted neighborhood density of a data point of the three-dimensional point cloud data is less than a first preset density, the data point is regarded as a noise point and removed. The calculation formula of the weighted neighborhood density of the data point is: ,in, Indicate point The weighted neighborhood density of Data points representing 3D point cloud data, Indicate point Neighborhood points, Represents the set of neighbor points within a preset radius, represents the distance attenuation coefficient, represents the distance attenuation weight coefficient, Indicate point and neighbors The Euclidean distance of

[0055] Step S204, using the Kalman filter method to smooth the position and speed of the target in the motion data; specifically, initialize the state vector of the target ,in, represents the state vector of the target, and Represent the current position and current speed of the target respectively. Based on the state transfer matrix, predict the position and speed of the target at the next moment. Combine the motion data and the predicted results, update the state vector, and smooth the target trajectory. The state transfer matrix is ​​expressed as: , A represents the state transfer matrix, represents the time step;

[0056] Step S205 , adding a unified timestamp to the processed optical image data, three-dimensional point cloud data and motion data, synchronizing the acquisition time of each data, and obtaining first feature data, second feature data and third feature data.

[0057] In one embodiment of the present invention, the specific steps of step S202 include:

[0058] Step S2021, using the k nearest neighbor algorithm, taking the Euclidean distance as the metric, to determine the k nearest neighbor points of each point in the three-dimensional point cloud data. Preferably, k is set to 10, and the formula of the Euclidean distance is: ,in Indicate point and The Euclidean distance between and Respectively indicate points and The x-axis coordinate of and Respectively indicate points and The y-axis coordinate of and Respectively indicate points and The z-axis coordinate of

[0059] Step S2022, fitting the neighboring points obtained in step S2021 by the least squares method to obtain a local plane, the plane equation is: , where x, y and z represent the coordinates of the point on the plane, a, b and c represent the three components of the plane normal vector, and d represents the offset of the plane;

[0060] Step S2023, obtaining the normal vector of the plane according to the plane equation .

[0061] In one embodiment of the present invention, an image feature extraction model is constructed according to an improved convolutional neural network, and the image feature extraction model is used to extract image texture features of the first feature data;

[0062] The image feature extraction model consists of an input layer, a multi-scale convolutional layer, a channel attention module, a multi-scale fusion layer, and an output layer;

[0063] The input layer is used to receive the first feature data;

[0064] The multi-scale convolution layer uses convolution kernels of different scales for feature extraction. Specifically, convolution kernels of different scales are used simultaneously in each convolution layer to extract and process the feature map, and the feature maps output by different convolution kernels are concatenated to obtain a multi-scale feature map.

[0065] The channel attention module is used to calculate the weight of each channel through global average pooling and perform weighted processing on the channels. The calculation formula of the channel attention module is: ,in, represents the feature map after channel attention weighting, represents a multi-scale feature map, and Represent the first weight matrix and the second weight matrix respectively, H and W represent the height and width of the multi-scale feature map respectively, h and w represent the index of the height and width respectively, c represents the index of the number of channels, sigmoid represents the sigmoid activation function, and ReLU represents the ReLU activation function;

[0066] The multi-scale fusion layer is used to concatenate the features extracted by the first half of the convolutional layer in the image feature extraction model with the features extracted by the second half of the convolutional layer in the channel dimension to obtain fused multi-scale features;

[0067] The output layer is used to reduce the dimension of multi-scale features through global average pooling and output image texture features.

[0068] In one embodiment of the present invention, a target candidate region is generated according to a self-attention mechanism, specifically comprising: obtaining an attention weight according to image texture features and a formula of the self-attention mechanism, and when it is determined that the attention weight is greater than a preset attention weight threshold, taking it as a candidate target region, and the formula of the self-attention mechanism is:

[0069] ;

[0070] in, represents the attention weight, express Activation function, , and Respectively represent query weight parameters, key weight parameters and value weight parameters, represents the image texture feature, T represents the transposition operation, and Respectively represent the horizontal distance and vertical distance between the mth position and the nth position, Represents the dimension of image texture features, Represents the minimum value between 0 and 1. express Activation function, represents the convolution operation, represents the local convolution kernel, The neighborhood area represents the texture features of the image.

[0071] In one embodiment of the present invention, the spatial geometric features of the target in the target candidate area are extracted according to the second feature data, and the specific steps include:

[0072] Step S301, dividing the second feature data into voxel grids of a fixed size;

[0073] Step S302: for each non-empty voxel, input the point set into PointNet to extract the voxel local features of the voxel; specifically, use a multi-layer perceptron to perform feature mapping on each point in the voxel, and aggregate the features of all points in the voxel through a maximum pooling operation to obtain the voxel local features, wherein the calculation formula of the feature mapping is: , Represents the features of the points obtained through feature mapping, represents the features of the points inside the voxel, represents a multi-layer perceptron;

[0074] Step S303, a three-dimensional convolution operation is performed on the local features of each voxel to obtain a spatial geometric feature. Specifically, the calculation formula of the three-dimensional convolution is: ,in, Represents the spatial geometric features obtained after three-dimensional convolution processing, represents the local feature of the voxel, r represents the receptive radius of the convolution kernel, p, q and t represent the index of the relative position of the convolution kernel on the local feature of the voxel, i, j and k represent the index of the center position of the convolution kernel in the convolution process, Represents the convolution kernel of a 3D convolution.

[0075] In one embodiment of the present invention, the image texture features and the spatial geometric features are fused and matched, and the specific steps include:

[0076] Step S401, projecting the second feature data to the target candidate area of ​​the first feature data to obtain the target projection area, and the projection calculation formula is: , where u and v represent the two-dimensional coordinate index of the projection position, and K represents the preset camera intrinsic parameter matrix. represents the preset camera extrinsic parameter matrix, x, y and z represent the three-dimensional coordinate index of the midpoint of the second feature data respectively;

[0077] Step S402, sampling in the image texture feature according to the target projection area to obtain the point texture feature corresponding to each point in the second feature data;

[0078] Step S403, concatenating the spatial geometric features of each point in the second feature data with the corresponding point texture features, and finally obtaining a target area containing image texture features and spatial geometric features.

[0079] In one embodiment of the present invention, image texture features and spatial geometric features are fused to obtain both the appearance attributes of the target and its spatial structure, thereby achieving complementary advantages. When the target motion trajectory changes rapidly or is suddenly occluded, the image texture features may become invalid, but the spatial geometric features can provide stable target structure information.

[0080] In one embodiment of the present invention, the motion vector of the pixel in the first feature data is calculated by an optical flow algorithm to generate an initial motion vector field, and the initial motion vector field is trajectory corrected by a Kalman filter algorithm using the third feature data to obtain a corrected motion vector field, wherein the calculation formula of the optical flow is: , by solving this formula, we can get the motion vector , , and Respectively represent the horizontal gradient, vertical gradient and time gradient of the pixel, and u and v represent the movement direction and speed of the pixel. The specific steps of trajectory correction of the initial motion vector field through the Kalman filter algorithm include: first defining the position and motion vector of the target, then predicting the state of the target at the next moment, and then using the motion vector field to update the predicted value, so as to correct the trajectory.

[0081] In one embodiment of the present invention, a motion feature vector is constructed for each motion vector of the corrected motion vector field, and the motion feature vector includes pixel coordinates, pixel motion direction and pixel speed; the corrected motion vector field is clustered using a K-means clustering algorithm to obtain K clusters, each cluster represents a motion pattern area, and the center point of each cluster represents the dynamic motion feature of the cluster, and the dynamic motion feature includes: average speed, average motion direction and motion pattern area range; wherein the calculation formula of the average motion direction is: , represents the average motion direction of the kth cluster, arctan represents the inverse tangent function, represents the velocity of the i-th motion vector, represents the moving direction angle of the i-th motion vector, sin and cos represent the sine function and cosine function respectively; the motion pattern area range is determined by the bounding box of the cluster.

[0082] In one embodiment of the present invention, the optical flow algorithm can extract pixel-level motion information from continuous image frames to generate a motion vector field, which can capture the movement direction and speed of the target. The motion vector field is corrected by preprocessed motion data, and errors caused by noise, occlusion or mismatching can be corrected, thereby improving the overall accuracy of dynamic motion features.

[0083] In one embodiment of the present invention, the image texture features and dynamic motion features of the target area are unified to the same feature dimension through a fully connected layer and spliced ​​to obtain fused features. The fused features are input into the first classifier to obtain a target classification result. The target classification result includes: target type and target dynamic behavior type. For example, target types include: vehicles, pedestrians, animals, etc., and target dynamic behavior types include: moving, stationary, approaching, moving away, etc.

[0084] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are protected by the present embodiment.

Claims

1. An unmanned aerial vehicle multi-target detection and tracking system, characterized in that: It includes data acquisition and preprocessing module, target detection module, motion trajectory dynamic modeling module and target classification module: A data acquisition and preprocessing module, used to acquire optical image data, three-dimensional point cloud data and motion data of the target, and preprocess the above data to obtain first feature data, second feature data and third feature data respectively; Among them, motion data includes the position, speed and direction of the target; A target detection module is used to extract image texture features according to the first feature data and generate a target candidate region according to a self-attention mechanism, extract spatial geometric features of the target in the target candidate region according to the second feature data, fuse and match the image texture features with the spatial geometric features, and obtain a target region containing image texture features and spatial geometric features; Wherein, an image feature extraction model is constructed according to an improved convolutional neural network, and the image feature extraction model is used to extract image texture features of the first feature data; The image feature extraction model consists of an input layer, a multi-scale convolutional layer, a channel attention module, a multi-scale fusion layer, and an output layer; The input layer is used to receive the first feature data; The multi-scale convolution layer uses convolution kernels of different scales for feature extraction. Specifically, convolution kernels of different scales are used simultaneously in each convolution layer to extract and process the feature map, and the feature maps output by different convolution kernels are concatenated to obtain a multi-scale feature map. The multi-scale fusion layer is used to concatenate the features extracted by the first half of the convolutional layer in the image feature extraction model with the features extracted by the second half of the convolutional layer in the channel dimension to obtain fused multi-scale features; The channel attention module is used to calculate the weight of each channel through global average pooling and perform weighted processing on the channels; The output layer is used to reduce the dimension of multi-scale features through global average pooling and output image texture features; The spatial geometric features of the target in the target candidate area are extracted according to the second feature data, and the specific steps include: Step S301, dividing the second feature data into voxel grids of a fixed size; Step S302: for each non-empty voxel, input the point set into PointNet to extract the voxel local features of the voxel; specifically, use a multi-layer perceptron to perform feature mapping on each point in the voxel, and aggregate the features of all points in the voxel through a maximum pooling operation to obtain the voxel local features; Step S303, performing a three-dimensional convolution operation on the voxel local features of each voxel to obtain spatial geometric features; A motion trajectory dynamic modeling module, used to generate a motion vector field according to the first feature data, correct the motion vector field according to the third feature data, and perform cluster analysis on the corrected motion vector field to obtain dynamic motion features with cluster classification results; The motion vectors of pixels in the first feature data are calculated by an optical flow algorithm to generate an initial motion vector field, and the initial motion vector field is trajectory corrected by a Kalman filter algorithm using the third feature data to obtain a corrected motion vector field, and a motion feature vector is constructed for each motion vector in the corrected motion vector field, the motion feature vector includes pixel coordinates, pixel motion direction and pixel speed; the corrected motion vector field is clustered using a K-means clustering algorithm to obtain K clusters, each cluster represents a motion pattern area, and the center point of each cluster represents the dynamic motion characteristics of the cluster; The target classification module is used to aggregate image texture features and dynamic motion features to generate target classification results.

2. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: The specific steps of the pretreatment include: Step S201, using the image frame of the optical image data Processing, in which represents the Gaussian function value, and Respectively represent the standard deviation in the horizontal and vertical directions, exp represents the exponential function, and represent the velocity attenuation coefficients in the horizontal and vertical directions, respectively; x and y represent the coordinates of the pixels of the optical image data, respectively; represents the smoothing coefficient, and Respectively represent the gradients of the image frame in the horizontal and vertical directions; Step S202, calculating the normal vector of each point of the three-dimensional point cloud data by constructing a plane equation, and splicing it behind the three-dimensional point cloud data; Step S203: when it is determined that the weighted neighborhood density of a data point of the three-dimensional point cloud data is less than a first preset density, the data point is regarded as a noise point and removed. The calculation formula of the weighted neighborhood density of the data point is: ,in, Indicate point The weighted neighborhood density of Data points representing 3D point cloud data, Indicate point Neighborhood points, Represents the set of neighbor points within a preset radius, represents the distance attenuation coefficient, represents the distance attenuation weight coefficient, Indicate point and neighbors The Euclidean distance of Step S204, using a Kalman filter method to smooth the position and speed of the target in the motion data; Step S205 , adding a unified timestamp to the processed optical image data, three-dimensional point cloud data and motion data, synchronizing the acquisition time of each data, and obtaining first feature data, second feature data and third feature data.

3. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: The calculation formula of the channel attention module is: ,in, represents the feature map after channel attention weighting, represents a multi-scale feature map, and Represent the first weight matrix and the second weight matrix respectively, H and W represent the height and width of the multi-scale feature map respectively, h and w represent the indexes of the height and width respectively, c represents the index of the number of channels, sigmoid represents the sigmoid activation function, and ReLU represents the ReLU activation function.

4. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: Generate target candidate regions according to the self-attention mechanism, specifically including: obtaining attention weights according to image texture features and the formula of the self-attention mechanism, and when it is determined that the attention weight is greater than a preset attention weight threshold, taking it as a candidate target region. The formula of the self-attention mechanism is: ； in, represents the attention weight, express Activation function, , and Respectively represent query weight parameters, key weight parameters and value weight parameters, represents the image texture feature, T represents the transposition operation, and Respectively represent the horizontal distance and vertical distance between the mth position and the nth position, Represents the dimension of image texture features, Represents the minimum value between 0 and 1. express Activation function, represents the convolution operation, represents the local convolution kernel, The neighborhood area represents the texture features of the image.

5. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: The image texture features are fused and matched with the spatial geometric features. The specific steps include: Step S401, projecting the second feature data onto the target candidate area of ​​the first feature data to obtain a target projection area; Step S402, sampling in the image texture feature according to the target projection area to obtain the point texture feature corresponding to each point in the second feature data; Step S403, concatenating the spatial geometric features of each point in the second feature data with the corresponding point texture features, and finally obtaining a target area containing image texture features and spatial geometric features.

6. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: The calculation formula of the optical flow algorithm is: , by solving this formula, we can get the motion vector , , and They represent the horizontal gradient, vertical gradient and time gradient of the pixel respectively, and u and v represent the movement direction and speed of the pixel.

7. The unmanned aerial vehicle multi-target detection and tracking system according to claim 1, characterized in that: The dynamic motion features include: average speed, average motion direction and motion pattern area range; wherein the calculation formula of the average motion direction is: , represents the average motion direction of the kth cluster, arctan represents the inverse tangent function, represents the velocity of the i-th motion vector, represents the moving direction angle of the i-th motion vector, sin and cos represent the sine function and cosine function respectively; the motion pattern area range is determined by the bounding box of the cluster.

8. The unmanned aerial vehicle multi-target detection and tracking system according to claim 7, characterized in that: The image texture features and dynamic motion features of the target area are unified to the same feature dimension through a fully connected layer and spliced ​​to obtain fused features. The fused features are input into the first classifier to obtain target classification results, which include: target type and target dynamic behavior type.

Images