Unmanned aerial vehicle recognition and tracking method and system based on multi-modal fusion and trajectory modeling

Abstract

The application provides a UAV recognition and tracking method and system based on multi-modal fusion and trajectory modeling, and relates to the technical field of target detection and tracking. In the application, infrared / visible light images and audio signals are synchronously collected in key areas of an airport. After time alignment, visual and acoustic features are respectively extracted through a deep network, and respective UAV category probabilities and candidate target frames are output. Then, according to the confidence of the audio and visual classification results, weights are adaptively assigned, category fusion is performed at the probability level, and a unified multi-modal fusion representation is constructed at the feature level, so that high recognition reliability is still maintained in small target, weak texture and strong noise scenes. On this basis, the multi-modal fusion feature sequence is taken as input, the position and speed of the UAV at each time step are constructed into a trajectory state sequence, and a sequence generation model outputs a complete trajectory under conditional constraints, so as to realize continuous tracking, occlusion interval completion and future trend prediction of the UAV motion.

Description

Technical Field

[0001] This invention relates to the field of target detection and tracking technology, and in particular to a method and system for UAV identification and tracking based on multimodal fusion and trajectory modeling. Background Technology

[0002] In the unique low-altitude security scenario of airports, existing airport drone control systems suffer from a series of prominent problems, including incomplete and inaccurate detection, unstable tracking, and slow coordination. On the one hand, traditional solutions rely heavily on single radar or video surveillance, which are severely affected by weather, lighting, small target size, and noise interference, making them prone to missed or false detections and difficult to reliably identify drones in complex environments. On the other hand, existing tracking methods often employ simple detection and filtering approaches, which can only passively track observed points and lack generative prediction capabilities for obstruction, missed detections, and future movement trends. This results in broken trajectories, delayed warnings, and difficulty in providing continuous and reliable spatiotemporal information support for airport command and countermeasure equipment.

[0003] Therefore, there is an urgent need for UAV identification and tracking methods and systems based on multimodal fusion and trajectory modeling to solve the above problems. Summary of the Invention

[0004] To address the aforementioned technical problems in related technologies, this invention proposes a method and system for UAV identification and tracking based on multimodal fusion and trajectory modeling.

[0005] In a first aspect, the present invention provides a method for identifying and tracking unmanned aerial vehicles (UAVs) based on multimodal fusion and trajectory modeling, comprising the following steps:

[0006] S1. Acquire continuous audio and video streams from key areas of the airport, and align the audio and video streams with their timestamps to obtain an audio frame sequence. With image frame sequence Where T is the total number of frames;

[0007] S2, audio frame sequence After standard acoustic preprocessing and acoustic feature encoding, the data is input into an audio classification head for category scoring and then normalized to obtain the audio category probability vector. ;

[0008] S3, Sequence of image frames After inputting the pre-trained visual feature extraction network into the visual classification head, the network performs category scoring and then normalizes the results to obtain the visual category probability vector. Then, the drone's position is obtained based on the detection head, and the estimated velocity of the drone is calculated based on the drone's position in adjacent frames to construct the drone's motion state vector. ;

[0009] S4. Based on the audio category probability vector With visual category probability vector The confidence level is adaptively weighted, and class fusion is performed at the probability level to obtain the fused class probability vector. And based on the fused class probability vector Construct a unified multimodal fusion feature vector at the feature level. Then, the multimodal fusion feature vectors are used. Projecting onto a unified latent space and passing through a multi-layer self-attention encoder, a holistic encoded sequence containing preceding and following temporal context information is obtained. ;

[0010] S5. Transfer the UAV motion state vector from the previous moment. Mapping and linear transformation yield the embedding vector and the single-frame coded latent vector. The linear transformation and summation yield the current frame's decoding input vector. Decode the input vector based on the current frame. The current trajectory hidden state representation is obtained by modeling with a multi-layer self-attention decoder, and then output as a UAV trajectory prediction sequence after linear mapping and training. .

[0011] Specifically, step S1 includes the following steps:

[0012] S11. Acquire a continuous audio stream using a microphone array deployed in the first critical area; the first critical area is the perimeter of the airport runway and the vicinity of the control tower.

[0013] S12. Acquire continuous video streams using infrared / visible light cameras deployed in the second key area; the second key area is the airport perimeter and apron location.

[0014] S13. Establish a unified discrete time axis to align the continuous audio stream and the continuous video stream to obtain an audio frame sequence. With image frame sequence .

[0015] Specifically, step S13 includes the following steps:

[0016] S131. Establish a unified discrete time axis;

[0017] S132. Discretize the continuous audio stream according to a unified discrete time axis and a fixed sampling rate to obtain the audio frame corresponding to each time step t. The audio frame sequence is obtained by stacking the audio frames of all time steps in order. ; The number of sampling points contained in each frame; dimension Indicates shared ownership Frame, per frame dimension;

[0018] S133. Extract the video image frames corresponding to each time step t from the continuous video stream according to a unified discrete time axis. The image frame sequence is obtained by stacking the video image frames from all time steps in order. ; where image frame sequence Represents the image sequence within the entire detection window; dimension Indicates shared ownership Frames, each frame is Pixels, H is the image height, and W is the image width.

[0019] Specifically, step S2 includes the following steps:

[0020] S21, audio frame sequence Each audio frame Acoustic maps are obtained by performing standard acoustic preprocessing. The standard acoustic preprocessing includes Short Time Fourier Transform (STFT), Mel filtering, and logarithmic compression.

[0021] S22, Acoustic diagram Audio features are obtained by inputting the pre-trained acoustic feature encoding network. ;

[0022] S23, The audio features Input audio classification head to get The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the audio category probability vector. .

[0023] Specifically, step S3 includes the following steps:

[0024] S31, Image frame sequence Each video image frame Visual features are obtained by inputting them into a pre-trained visual feature extraction network. :

[0025] S32, the visual features Input the classification branch of the detection head of the pre-trained visual feature extraction network to obtain The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the visual category probability vector. ;

[0026] S33. Select the candidate box with the highest confidence level that belongs to the category of UAV from the multiple candidate boxes output by the regression branch of the detection head of the pre-trained visual feature extraction network as the UAV target detection box. :

[0027] ;

[0028] in, Let be the target detection bounding box for the drone, representing the th . The bounding box parameter vector of the UAV target in the frame; In the first Normalized coordinates of the center of the UAV target detection box in the frame; These are the normalized width and height of the drone target detection bounding box, respectively; top right corner. Indicates transpose;

[0029] S34. Calculate the estimated velocity of the UAV based on the position of the UAV in adjacent frames, and construct the UAV motion state vector based on the UAV's position and estimated velocity. ;

[0030] The estimated velocity of the drone, calculated based on the drone's position in adjacent frames, is shown below:

[0031] ;

[0032] in, For the first Estimated velocities of the frame drone in the horizontal and vertical directions; The time interval between two adjacent frames; The coordinates are the normalized coordinates of the center of the UAV target detection box in frame t-1; the UAV position is the center coordinate of the UAV target detection box.

[0033] By concatenating the drone's position with its estimated velocity, the first... UAV motion state vector of a frame :

[0034] ;

[0035] in, For the state dimension; top right corner This indicates transpose.

[0036] Specifically, step S4 includes the following steps:

[0037] S41. Obtain the audio category probability vector. The highest probability component is used as the overall audio confidence of the current frame. ;

[0038] S42, Obtain the visual category probability vector The highest probability component is used as the overall visual confidence of the current frame. ;

[0039] S43, Overall audio confidence Overall visual confidence Linear normalization to audio fusion weights Visual fusion weights ;

[0040] S44. Based on audio fusion weights Visual fusion weights audio category probability vector and visual category probability vector The weighted summation yields the fused class probability vector. :

[0041] ;

[0042] Where C represents the total number of categories;

[0043] S45. Concatenate the visual features, weighted audio features, and fused drone category probabilities into a multimodal fusion feature vector. :

[0044] ;

[0045] in, Visual features; Audio features; For audio fusion weights; The probability of drone category after fusion; This represents a vector concatenation operation by dimension; Represents the fused feature vector Total dimensions; For audio feature dimensions; For visual feature dimensions;

[0046] S46. Fusing multimodal feature vectors through linear transformation Projecting onto the unified latent space yields the single-frame coded latent vector. ;

[0047] S47. Encode the latent vector of each frame of the entire time series. After stacking, the encoder input sequence matrix is ​​obtained. Input a multi-layer self-attention encoder to obtain a holistic encoded sequence that includes preceding and following temporal context information. .

[0048] Specifically, step S5 includes the following steps:

[0049] S51. Transfer the UAV motion state vector from the previous moment. Mapping to the same latent space as a multi-layer self-attention encoder yields the predicted state from the previous time step. ;

[0050] S52. Transform the predicted state from the previous time step using a linear transformation. Projected to The embedding vector of the predicted state in the previous time step is obtained from the latent space. ;

[0051] S53. Embed the vector of the predicted state from the previous time step. With single-frame encoded latent vectors The linear transformation and summation yield the decoding input vector for the current frame. ;

[0052] S54. Utilize a multi-layer self-attention decoder to decode the historical sequence of the input vector. With the overall coding sequence Modeling is performed to obtain the hidden state representation of the trajectory at the current time. ;

[0053] S55. Representing the hidden state of the trajectory through linear mapping. Convert to predicted trajectory state vector And it is trained using a trajectory tracking loss function;

[0054] The predicted trajectory state vector The expression is as follows:

[0055] ;

[0056] in, For the first The predicted drone location in the frame; The first The frame predicts the drone's speed in the horizontal and vertical directions; This is the output layer weight matrix; This is the output layer bias vector; For the state dimension; top right corner Indicates transpose;

[0057] The trajectory tracking loss function uses the mean squared error to supervise the entire trajectory, and its expression is as follows:

[0058] ;

[0059] in, For trajectory tracking loss; It is the Euclidean norm, used to measure the distance between the true state and the predicted state; For drones in the The state vector of a frame;

[0060] S56, Output the predicted trajectory state vector sequence As a sequence for predicting drone trajectories.

[0061] Specifically, the pre-trained acoustic feature encoding network mentioned in step S22 is a convolutional-recurrent hybrid network, consisting of several convolutional layers and recurrent layers, with the input acoustic map... The output is audio features. This is achieved through prior training.

[0062] Specifically, the pre-trained visual feature extraction network mentioned in step S31 is an improved YOLO backbone network, which includes convolutions, feature pyramids, and a backbone structure, and takes video image frames as input. The output is visual features. This is achieved through prior training.

[0063] Secondly, the present invention provides a UAV identification and tracking system based on multimodal fusion and trajectory modeling, which, based on the UAV identification and tracking method based on multimodal fusion and trajectory modeling described in the first aspect above, includes the following units:

[0064] This invention provides a method for UAV identification and tracking based on multimodal fusion and trajectory modeling, including:

[0065] The acquisition module is used to acquire continuous audio and video streams from key areas of the airport, and then align the time of the continuous audio and video streams to obtain an audio frame sequence. With image frame sequence Where T is the total number of frames;

[0066] The audio category probability extraction module is used to extract audio frame sequences. After standard acoustic preprocessing and acoustic feature encoding, the data is input into an audio classification head for category scoring and then normalized to obtain the audio category probability vector. ;

[0067] The drone motion state extraction module is used to extract image frame sequences. After inputting the pre-trained visual feature extraction network into the visual classification head, the data is scored and then normalized to obtain the visual class probability vector. Then, the drone's position is obtained based on the detection head, and the estimated velocity of the drone is calculated based on the drone's position in adjacent frames to construct the drone's motion state vector. ;

[0068] The overall encoding sequence construction module is used to construct the sequence based on the audio category probability vector. With visual category probability vector The confidence level is adaptively weighted, and class fusion is performed at the probability level to obtain the fused class probability vector. And based on the fused class probability vector Construct a unified multimodal fusion feature vector at the feature level. Then, the multimodal fusion feature vectors are used. Projecting onto a unified latent space and passing through a multi-layer self-attention encoder, a holistic encoded sequence containing preceding and following temporal context information is obtained. ;

[0069] The UAV trajectory prediction output module is used to output the UAV's motion state vector from the previous moment. Mapping and linear transformation yield the embedding vector and the single-frame coded latent vector. The linear transformation and summation yield the current frame's decoding input vector. Decode the input vector based on the current frame. The current trajectory hidden state representation is obtained by modeling with a multi-layer self-attention decoder, and then output as a UAV trajectory prediction sequence after linear mapping and training. .

[0070] The UAV identification and tracking method based on multimodal fusion and trajectory modeling provided by this invention uses multimodal adaptive fusion identification: it proposes a time alignment and confidence adaptive weighting mechanism for audio-infrared / visible light images, and performs dynamic fusion at both the category probability layer and the feature layer. Compared with single sensor or fixed weight fusion, it significantly improves the stability of UAV identification under strong noise, weak imaging and small target conditions, and improves the stable identification capability of UAVs under complex conditions such as noise interference, bad weather and small targets.

[0071] Furthermore, the UAV identification and tracking method based on multimodal fusion and trajectory modeling provided by this invention adopts position + velocity-oriented generative trajectory modeling: the temporal features after multimodal fusion are directly mapped into a trajectory state sequence containing "normalized position + velocity", and the complete trajectory is output by autoregression using a sequence generation model, thereby realizing continuous tracking of UAV motion, completion of occlusion intervals and prediction of future trends, breaking through the limitations of traditional "detection + filtering" which is prone to trajectory breakage and difficult to predict.

[0072] This invention, starting from the practical needs of low-altitude airport security, proposes to improve the robust identification of UAV targets in complex environments by using multimodal fusion. It also introduces generative trajectory modeling to continuously estimate the position and speed of UAVs, complete occlusions, and predict trends, thus connecting the technical links of "multi-source perception - intelligent identification - trajectory tracking". This enables airports to build a more reliable and timely UAV intrusion monitoring and response capability, solving a series of prominent problems in the existing airport UAV prevention and control, such as "incomplete detection, inaccurate detection, unstable tracking, and slow linkage". Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A schematic diagram of a UAV identification and tracking method based on multimodal fusion and trajectory modeling provided in an embodiment of the present invention;

[0075] Figure 2 A schematic diagram of a drone identification and tracking system based on multimodal fusion and trajectory modeling provided in an embodiment of the present invention. Detailed Implementation

[0076] The present invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0077] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0078] Example 1

[0079] refer to Figure 1 This embodiment provides a UAV identification and tracking method based on multimodal fusion and trajectory modeling, including the following steps:

[0080] S1. Acquire continuous audio and video streams from key areas of the airport, and align the audio and video streams with their timestamps to obtain an audio frame sequence. With image frame sequence ;

[0081] S11. Acquire a continuous audio stream using a microphone array deployed in the first critical area; the first critical area is the perimeter of the airport runway and the vicinity of the control tower.

[0082] Continuous audio streams are collected by microphone arrays deployed at locations such as the perimeter of the airport runway and near the control tower.

[0083] S12. Acquire continuous video streams using infrared / visible light cameras deployed in the second key area; the second key area is the airport perimeter and apron location.

[0084] Continuous video streams are captured by infrared / visible light cameras deployed at airport perimeters and tarmac locations;

[0085] S13. Establish a unified discrete time axis to align the continuous audio stream and the continuous video stream to obtain an audio frame sequence. With image frame sequence ;

[0086] S131. Establish a unified discrete time axis;

[0087] On this unified discrete time axis, at fixed time intervals Audio and images are sampled synchronously to form a discrete time series of length T. It is stipulated that at each time step t, there must be an audio frame and an image frame at the same time to facilitate subsequent multimodal joint modeling. The time interval (scalar) between adjacent time steps, in seconds, is fixed throughout the system. This represents the number of frames (scalar) within the current detection window, i.e., the total number of frames.

[0088] S132. Discretize the continuous audio stream according to a unified discrete time axis and a fixed sampling rate to obtain the audio frame corresponding to each time step t. The audio frame sequence is obtained by stacking the audio frames of all time steps in order. ;

[0089] After the continuous audio stream is discretized at a fixed sampling rate (sliced ​​within a detection window with a fixed window length and a fixed sliding step size), the t-th discrete moment of the continuous audio stream corresponds to an audio frame. ,in For the first Frame audio sample vector; The number of sampling points (scalar) contained in each frame; dimension Indicates shared ownership Frame, per frame dimension;

[0090] S133. Extract the video image frames corresponding to each time step t from the continuous video stream according to a unified discrete time axis. Ensure that it matches the audio frame. With consistent timestamps, video image frames from all time steps are stacked sequentially to obtain an image frame sequence. ; where image frame sequence Represents the image sequence within the entire detection window; dimension Indicates shared ownership Frames, each frame is Pixel.

[0091] On the same timeline as the audio, the t-th discrete moment of a continuous video stream corresponds to one frame. ,in Let H be the video image frame corresponding to the t-th time step; H is the image height (in pixels, scalar); W is the image width (in pixels, scalar).

[0092] This also includes, if no matching audio frame is found in the continuous video stream Video image frames with consistent timestamps are aligned using interpolation.

[0093] If a modality's data is missing or its timestamp is abnormal at certain times, it can be corrected using simple rules (such as discarding that time step or interpolating with adjacent frames) to ensure that the processed sequence still maintains its length. And ensure each time step It also has one frame of audio. With a frame of image .

[0094] S2, audio frame sequence After standard acoustic preprocessing and acoustic feature encoding, the data is input into an audio classification head for category scoring and then normalized to obtain the audio category probability vector. ;

[0095] S21, audio frame sequence Each audio frame Acoustic maps are obtained by performing standard acoustic preprocessing. ;

[0096] First, analyze each audio frame in the audio frame sequence. Perform standard acoustic preprocessing (including Short Time Fourier Transform (STFT), Mel filtering, and logarithmic compression) to convert the time-domain waveform into a time-frequency acoustic graph. .

[0097] S22, Acoustic diagram Audio features are obtained by inputting the pre-trained acoustic feature encoding network. ;

[0098] ;

[0099] in, For pre-trained acoustic feature encoding networks; For the first The audio features of a frame are represented by high-dimensional audio features; The audio feature dimension (scalar) is fixed throughout the entire process;

[0100] The pre-trained acoustic feature encoding network is a CRNN (Convolutional-Recurrent Hybrid Network), consisting of several convolutional and recurrent layers, with the input acoustic map... The output is audio features. Through prior training, the overall [performance / ability] will be [improved / successful]. ;

[0101] S23, The audio features Input audio classification head to get The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the audio category probability vector. ;

[0102] The process is shown in the following formula:

[0103] ;

[0104] in, This is an audio classification head, used for audio classification mapping. It typically consists of several fully connected layers with an input dimension of [missing information]. The output dimension is ; Indicates the first The probability vector of which audio frames belong to each category; Indicates the first The audio frame belongs to the first The probability of a class (e.g., drone, commercial aircraft, birds, background, etc.); The total number of categories (scalar) remains consistent across both audio and visual modalities; For a normalized exponential function, for a length of The vector is normalized after exponentiation of each element, so that the output components are non-negative and sum to 1.

[0105] In subsequent multimodal fusion, the overall audio confidence level will be used, such as the maximum component frequency. The probability component corresponding to the "drone" category (denoted as ); ,for (One dimension in it).

[0106] S3, Sequence of image frames After inputting the pre-trained visual feature extraction network into the visual classification head, the data is scored and then normalized to obtain the visual class probability vector. Then, the drone's position is obtained based on the detection head, and the estimated velocity of the drone is calculated based on the drone's position in adjacent frames to construct the drone's motion state vector. ;

[0107] S31, Image frame sequence Each video image frame Visual features are obtained by inputting them into a pre-trained visual feature extraction network. :

[0108] ;

[0109] in, For pre-trained visual feature extraction networks; For the first Visual feature vectors of a frame image; This is the visual feature dimension (scalar), which will be concatenated with audio features in subsequent multimodal fusion.

[0110] The pre-trained visual feature extraction network is an improved YOLO backbone network, which includes structures such as convolutions, feature pyramids, and a backbone, and takes video image frames as input. The output is visual features. This is achieved through prior training. ;

[0111] S32, the visual features Input the classification branch of the detection head of the pre-trained visual feature extraction network to obtain The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the visual category probability vector. ;

[0112] The process is shown in the following formula:

[0113] ;

[0114] in, This is the visual classification head, used for visual classification mapping. It is the classification branch of the YOLO detection head, with an input dimension of [missing information]. The output dimension is ; Let be the probability vector of the t-th video image frame belonging to each category; For the video image frame belonging to frame t, the... The probability of a class (such as drones, commercial aircraft, birds, background, etc.); The total number of categories (scalar, consistent with C in audio branch S2); For a normalized exponential function, for a length of The vector is normalized after exponentiation of each element, so that the output components are non-negative and sum to 1.

[0115] The YOLO detection head here is not a separate independent network, but a detection sub-network built on the feature map output by the pre-trained visual feature extraction network. Together with the feature extraction backbone, it constitutes the pre-trained visual feature extraction network.

[0116] Steps S31-S32 are the process of visual feature extraction and multi-class probability estimation;

[0117] It is understood that in this embodiment, the category of c includes drones, civil aircraft, birds, and background; it can be changed according to actual needs.

[0118] S33. Select the candidate box with the highest confidence level that belongs to the category of UAV from the multiple candidate boxes output by the regression branch of the detection head of the pre-trained visual feature extraction network as the UAV target detection box. :

[0119] ;

[0120] in, Let be the target detection bounding box for the drone, representing the th . The bounding box parameter vector of the UAV target in the frame; In the first The normalized coordinates of the center of the UAV target detection box in the frame (relative to the image width and height, usually in the range [0,1]); These are the normalized width and height of the drone target detection bounding box, respectively (also in...) (Interval); upper right corner Indicates transpose;

[0121] Understandably, the YOLO detection head includes a classification branch and a regression branch, and the visual classification head in step S32... This actually corresponds to the classification branch in the YOLO detection head of the pre-trained visual feature extraction network (the improved YOLO backbone network), used for output. The multiple candidate boxes output by the detection head mentioned in step S33 are the outputs of the regression branch in the same YOLO detection head, used to obtain the UAV target detection boxes. Therefore, the YOLO detection head described in S32 and S33 has the same structure; the only difference is that its classification and regression functions are used in the descriptions, respectively.

[0122] S34. Calculate the estimated velocity of the UAV based on the position of the UAV in adjacent frames, and construct the UAV motion state vector based on the UAV's position and estimated velocity. The location of the UAV is the center coordinate of the UAV target detection box;

[0123] For subsequent generative trajectory tracking, only the center coordinates of the UAV target detection box are selected to construct a 2D position as the UAV's position, and the estimated velocity of the UAV is calculated based on the position of the UAV in adjacent frames.

[0124] The estimated velocity of the drone, calculated based on the drone's position in adjacent frames, is shown below:

[0125] ;

[0126] in, For the first Estimated velocities of the frame drone in the horizontal and vertical directions; The time interval between two adjacent frames; In the first The normalized coordinates of the center of the drone target detection box in the frame, which is the position of the drone in the current frame; The normalized coordinates of the center of the UAV target detection box in frame t-1 are the positions of the UAV in the adjacent frames of the current frame.

[0127] For the first frame , can make Alternatively, neighboring frame estimation can be used; the specific implementation can be flexibly chosen.

[0128] By concatenating the drone's position with its estimated velocity, the first... UAV motion state vector of a frame :

[0129]

[0130] in, For drones in the The state vector of a frame (position + velocity); The state dimension is fixed at 4, and it is directly used as the supervision and prediction target in subsequent generative trajectory decoding; top right corner Indicates transpose;

[0131] S4. Based on the audio category probability vector With visual category probability vector The confidence level is adaptively weighted, and class fusion is performed at the probability level to obtain the fused class probability vector. And based on the fused class probability vector Construct a unified multimodal fusion feature vector at the feature level. Then, the multimodal fusion feature vectors are used. Projecting onto a unified latent space and passing through a multi-layer self-attention encoder, a holistic encoded sequence containing preceding and following temporal context information is obtained. ;

[0132] S41. Obtain the audio category probability vector. The maximum probability component is used as the overall audio confidence of the current frame. Its expression is ;

[0133] Understandable Indicates the first The probability vector of which audio frames belong to each category; Indicates the first The audio frame belongs to the first The probability of a class (e.g., drone, commercial aircraft, birds, background, etc.); Total number of categories; audio category probability vector The highest probability component is the th The audio frame belongs to the first The maximum value among the probabilities of a class;

[0134] S42, Obtain the visual category probability vector The highest probability component is used as the overall visual confidence of the current frame. Its expression is ;

[0135] Similarly, Let be the probability vector of the t-th video image frame belonging to each category; For the video image frame belonging to frame t, the... The probability of a class (e.g., drone, commercial aircraft, birds, background, etc.); Total number of categories; visual category probability vector The highest probability component is the th Frame by frame of video image belongs to the first The maximum value among the probabilities of a class;

[0136] S43, Overall audio confidence Overall visual confidence Linear normalization to audio fusion weights Visual fusion weights As shown in the following formula:

[0137] ;

[0138] in: The first Frame audio fusion weights and visual fusion weights; always satisfy When the confidence level of a certain mode is low, its corresponding weight is automatically reduced.

[0139] S44. Based on audio fusion weights Visual fusion weights audio category probability vector and visual category probability vector The weighted summation yields the fused class probability vector. Its expression is as follows:

[0140] ;

[0141] in, For the first The class probability vector after frame fusion This indicates that after fusion, it belongs to the first... The probability of classes (including the "drone" class); compared to single-modality, The sensitivity to single-mode noise is reduced.

[0142] S45. Concatenate the visual features, weighted audio features, and fused drone category probabilities into a multimodal fusion feature vector. :

[0143] ;

[0144] in, Visual features; Audio features; For audio fusion weights; The probability of the drone category after fusion (scalar); This represents a vector concatenation operation by dimension; Represents the fused feature vector The total dimension (scalar).

[0145] Multimodal fusion feature vector Visual information, weighted audio information, and the probability of drone presence were collected at each time step, which served as direct input to the subsequent encoder.

[0146] S46. Fusing multimodal feature vectors through linear transformation Projecting onto the unified latent space yields the single-frame coded latent vector. As shown in the following formula:

[0147] ;

[0148] in, For the first Frame-coded input latent vector; The weight matrix is ​​a linear mapping. It is the bias vector; This is the encoder's hidden layer dimension (scalar), which will remain consistent in the S5 decoder. This step involves fusing the multimodal feature vectors. Linear embedding into the encoder latent space;

[0149] S47. Encode the latent vector of each frame of the entire time series. After stacking, the input is fed into a multi-layer self-attention encoder to obtain the overall encoded sequence containing preceding and following temporal context information. ;

[0150] Encode the latent vector of each frame of the entire time series. After stacking, the encoder input sequence matrix is ​​obtained. Then the encoder is input into the sequence matrix. Input multilayer self-attention encoder Obtain the overall encoded sequence containing preceding and following temporal context information. Its expression is shown below;

[0151] ;

[0152] Among them, the multilayer self-attention encoder It is an encoding function based on the Transformer self-attention structure; Represents the parameter set; the overall encoded sequence The encoder outputs a sequence matrix, with each row corresponding to the encoded representation of a time step;

[0153] In subsequent steps, the overall encoded sequence And its single-frame representation The "memory" used as a decoder, along with the jump connection injection information, is used to generate the drone's trajectory state.

[0154] S5. Transfer the UAV motion state vector from the previous moment. Mapping and linear transformation yield the embedding vector and the single-frame coded latent vector. The linear transformation and summation yield the current frame's decoding input vector. Decode the input vector based on the current frame. The current trajectory hidden state representation is obtained by modeling with a multi-layer self-attention decoder, and then output as a UAV trajectory prediction sequence after linear mapping and training. ;

[0155] S51. Transfer the UAV motion state vector from the previous moment. Mapping to the same latent space as a multi-layer self-attention encoder yields the predicted state from the previous time step. ;

[0156] In the training phase, a teacher forcing strategy is typically employed. The inference phase uses the prediction results from the previous step of the model: ;

[0157] From step S3, we can see that... For drones in the The frame's state vector (position + velocity) represents the drone's motion state vector and can be used to represent the drone's motion state vector at the current moment. The state dimension is fixed at 4. Let be the true state vector (position + velocity) of the UAV in frame t-1, and let represent the motion state vector of the UAV in the previous moment.

[0158] S52. Transform the predicted state from the previous time step using a linear transformation. Projected to The embedding vector of the predicted state in the previous time step is obtained from the latent space. Its expression is: ;

[0159] in, Let be the embedding vector from the predicted state at the previous time step in the decoding input of frame t; To decode the input weight matrix; To decode the input bias vector;

[0160] This step is to map the trajectory state from the previous time step to the same latent space as the encoder, providing a temporal prior for the decoder.

[0161] S53. Embed the vector of the predicted state from the previous time step. With single-frame encoded latent vectors The linear transformation and summation yield the decoding input vector for the current frame. ;in, The expression is as follows:

[0162] ;

[0163] in, is the decoding input vector of the t-th frame, which is the encoding input vector of the current frame, combining the trajectory state of the previous time step and the multimodal observation of the current time step; This is a jump-link mapping matrix used to map the latent vectors of a single frame. Mapping to The space of addition, ;

[0164] This step combines the "previous time state information" and the "current time multimodal coding information" to form a conditional model of the trajectory at the current time. This "time alignment + jump connection" structure makes the trajectory generation process more consistent with the current frame perception information.

[0165] S54. Utilize a multi-layer self-attention decoder to decode the historical sequence of the input vector. With the overall coding sequence Modeling is performed to obtain the hidden state representation of the trajectory at the current time. The expression is as follows:

[0166] ;

[0167] in, It is a multi-layer self-attention decoder, which is a multi-layer Transformer decoder. This represents the parameter set, which is shared or partially shared with the multilayer self-attention encoder (the same). gather); It contains a self-attention layer for modeling. Time dependence; Internally, it contains an encoder-decoder attention layer, utilizing... As a memory; From frame 1 to frame 2 The sequence of all decoded input vectors in a frame represents the historical sequence of decoded input vectors; For the decoder in the The hidden state representation of a frame integrates historical trajectory information and multimodal observations to represent the hidden state representation of the trajectory at the current moment;

[0168] The multi-layer self-attention encoder in step S51 is the same as the multi-layer self-attention encoder in step S47. The multi-layer self-attention encoder is responsible for globally encoding and outputting the multi-modal fusion feature sequence. The multi-layer self-attention decoder in this step, based on the previous time step's state vector, performs autoregressive modeling of the trajectory sequence and predicts the current time step's hidden state representation of the trajectory. .

[0169] S55. Representing the hidden state of the trajectory through linear mapping. Convert to predicted trajectory state vector And it is trained using a trajectory tracking loss function;

[0170] Predicted trajectory state vector The expression is as follows:

[0171] ;

[0172] Among them, the predicted trajectory state vector Characterized the first The predicted position and velocity of the UAV in the frame are used as the predicted trajectory state vector; For the first The predicted drone location in the frame; The first The frame predicts the drone's speed in the horizontal and vertical directions; This is the output layer weight matrix; Output layer bias vector; linear mapping A linear mapping function for the linear output layer; linear mapping It can be seen as being caused by and A linear transformation is defined as shown in the formula above, that is, a linear mapping. This mapping represents the hidden state of the trajectory. Convert to predicted trajectory state vector .

[0173] The trajectory tracking loss function uses the mean squared error to supervise the entire trajectory, and its expression is as follows:

[0174] ;

[0175] in, Trajectory tracking loss (scalar); It is the Euclidean norm, used to measure the distance between the true state and the predicted state; For drones in the The state vector of a frame represents the actual state; This is the predicted trajectory state vector, representing the predicted state (output of this step).

[0176] S56, Output the predicted trajectory state vector sequence As a sequence for predicting drone trajectories.

[0177] On the timeline The final output of this step is the sequence of predicted trajectory state vectors:

[0178]

[0179] Predict the trajectory state vector sequence As a drone trajectory prediction sequence, this prediction sequence can be used for drone trajectory visualization and intrusion path analysis, as well as trajectory completion during periods of obstruction or weak signal.

[0180] The UAV identification and tracking method based on multimodal fusion and trajectory modeling provided in this embodiment uses multimodal adaptive fusion identification: it proposes a time alignment and confidence adaptive weighting mechanism for audio-infrared / visible light images, and performs dynamic fusion at both the category probability layer and the feature layer. Compared with single sensor or fixed weight fusion, it significantly improves the stability of UAV identification under strong noise, weak imaging and small target conditions, and improves the stable identification capability of UAVs under complex conditions such as noise interference, bad weather and small targets.

[0181] Furthermore, the UAV identification and tracking method based on multimodal fusion and trajectory modeling provided in this embodiment adopts position + velocity-oriented generative trajectory modeling: the temporal features after multimodal fusion are directly mapped into a trajectory state sequence containing "normalized position + velocity", and the complete trajectory is output by autoregression using a sequence generation model, thereby realizing continuous tracking of UAV motion, completion of occlusion intervals and prediction of future trends, breaking through the limitations of traditional "detection + filtering" which is prone to trajectory breakage and difficult to predict.

[0182] This embodiment, starting from the practical needs of low-altitude security at airports, proposes to improve the robust identification of UAV targets in complex environments by using multimodal fusion. It also introduces generative trajectory modeling to continuously estimate the position and speed of UAVs, complete occlusions, and predict trends, thus connecting the technical links of "multi-source perception - intelligent identification - trajectory tracking". This enables airports to build a more reliable and timely UAV intrusion monitoring and response capability, solving a series of prominent problems in the existing airport UAV prevention and control, such as "incomplete detection, inaccurate detection, unstable tracking, and slow linkage".

[0183] Example 2

[0184] refer to Figure 2 This embodiment provides a UAV identification and tracking system based on multimodal fusion and trajectory modeling, which is based on the UAV identification and tracking method based on multimodal fusion and trajectory modeling described in Embodiment 1, and includes the following modules:

[0185] The acquisition module is used to acquire continuous audio and video streams from key areas of the airport, and then align the time of the continuous audio and video streams to obtain an audio frame sequence. With image frame sequence Where T is the total number of frames;

[0186] The audio category probability extraction module is used to extract audio frame sequences. After standard acoustic preprocessing and acoustic feature encoding, the data is input into an audio classification head for category scoring and then normalized to obtain the audio category probability vector. ;

[0187] The drone motion state extraction module is used to extract image frame sequences. After inputting the pre-trained visual feature extraction network into the visual classification head, the data is scored and then normalized to obtain the visual class probability vector. Then, the drone's position is obtained based on the detection head, and the estimated velocity of the drone is calculated based on the drone's position in adjacent frames to construct the drone's motion state vector. ;

[0188] The overall encoding sequence construction module is used to construct the sequence based on the audio category probability vector. With visual category probability vector The confidence level is adaptively weighted, and class fusion is performed at the probability level to obtain the fused class probability vector. And based on the fused class probability vector Construct a unified multimodal fusion feature vector at the feature level. Then, the multimodal fusion feature vectors are used. Projecting onto a unified latent space and passing through a multi-layer self-attention encoder, a holistic encoded sequence containing preceding and following temporal context information is obtained. ;

[0189] The UAV trajectory prediction output module is used to output the UAV's motion state vector from the previous moment. Mapping and linear transformation yield the embedding vector and the single-frame coded latent vector. The linear transformation and summation yield the current frame's decoding input vector. Decode the input vector based on the current frame. The current trajectory hidden state representation is obtained by modeling with a multi-layer self-attention decoder, and then output as a UAV trajectory prediction sequence after linear mapping and training. .

[0190] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A method for UAV identification and tracking based on multi-modal fusion and trajectory modeling, characterized in that, The method comprises the following steps: S1, collect continuous audio stream and continuous video stream of key area of airport, and get audio frame sequence after aligning time of continuous audio stream and continuous video stream with image frame sequence ; wherein, T is total frame number; S2, the audio frame sequence After the standard acoustic pre-processing, the input audio classification head is encoded by the acoustic feature, and the category score is normalized to obtain the audio category probability vector ; S3, constructing a sequence of image frames After inputting the pre-trained visual feature extraction network, inputting the visual classification head to obtain the visual category probability vector after classifying and normalizing Then, the position of the unmanned aerial vehicle is obtained according to the detection head, and the estimated speed of the unmanned aerial vehicle is calculated according to the positions of adjacent frames of the unmanned aerial vehicle to construct a motion state vector of the unmanned aerial vehicle ; S4, assigning weights to the audio category probability vector and the visual category probability vector according to their confidence levels, fusing the categories at the probability level to obtain a fused category probability vector and fusing the categories at the feature level to obtain a unified multimodal fusion feature vector according to the fused category probability vector ; then projecting the unified multimodal fusion feature vector to a unified hidden space and passing it through a multi-layer self-attention encoder to obtain an overall encoding sequence containing front and back temporal context information ; S5, mapping and linearly transforming the last time's UAV motion state vector embedding vector and single-frame encoding hidden vector linearly transforming and adding to obtain the current frame decoding input vector , according to the current frame decoding input vector linearly mapping and training after modeling the current trajectory hidden state representation through the multi-layer self-attention decoder to output the UAV trajectory prediction sequence .

2. The method of claim 1, wherein, The step S1 specifically comprises the following steps: S11, collecting continuous audio stream through microphone array arranged in the first key area; the first key area is the perimeter of the airport runway, the position near the tower; S12, collecting continuous video stream through infrared / visible light camera arranged in the second key area; the second key area is the perimeter of the airport, the position of the parking apron; S13, establishing a unified discrete time axis to align the continuous audio stream and the continuous video stream to obtain an audio frame sequence with the image frame sequence .

3. The method of claim 2, wherein, The step S13 specifically comprises the following steps: S131, establishing a unified discrete time axis; S132, discretize the continuous audio stream to get audio frames corresponding to each time step t according to the uniform discrete time axis at a fixed sampling rate , stack the audio frames of all time steps in order to get an audio frame sequence ; , the number of sampling points contained in each frame; dimension , the total number of frames , the number of frames , dimension S133. Extract the video image frames corresponding to each time step t from the continuous video stream according to a unified discrete time axis. The image frame sequence is obtained by stacking the video image frames from all time steps in order. ; where image frame sequence Represents the image sequence within the entire detection window; dimension Indicates shared ownership Frames, each frame is Pixels, H is the image height, and W is the image width.

4. The method of claim 1, wherein, The step S2 specifically comprises the following steps: S21, obtaining a sequence of audio frames each audio frame obtaining an acoustic map by standard acoustic pre-processing ; the standard acoustic pre-processing comprises short-time Fourier transform (STFT), Mel filtering and log compression; S22, the acoustic map is input into a pre-trained acoustic feature encoding network to obtain audio features inputting the pre-trained acoustic feature encoding network to obtain audio features ; S23, The audio features Input audio classification head to get The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the audio category probability vector. .

5. The method of claim 1, wherein, The step S3 specifically comprises the following steps: S31, obtaining a sequence of image frames each video image frame inputting the pre-trained visual feature extraction network to obtain visual features : S32, the visual features Input the classification branch of the detection head of the pre-trained visual feature extraction network to obtain The unnormalized scores for each category are obtained, and then normalized using the softmax function to obtain the visual category probability vector. ; S33, select a candidate box with the highest confidence and the class of unmanned aerial vehicle from multiple candidate boxes output by a regression branch of a detection head of the pre-trained visual feature extraction network as an unmanned aerial vehicle target detection box : ； wherein, is a bounding box parameter vector of the drone target in the i-th frame; is a bounding box parameter vector of the drone target in the i-th frame; is a normalized coordinate of the center of the drone target bounding box in the i-th frame; is a normalized coordinate of the center of the drone target bounding box in the i-th frame; are the normalized width and height of the drone target bounding box, respectively; and denotes the transpose. S34, calculate an estimated velocity of the UAV based on the position of the adjacent frame UAV, and construct a UAV motion state vector according to the position of the UAV and the estimated velocity of the UAV ; The estimated speed of the unmanned aerial vehicle based on the position of the adjacent frame is as follows: ； wherein, is the first estimated speed of the drone in horizontal and vertical directions; is the time interval between two adjacent frames; is the normalized coordinate of the center of the drone target detection frame in the t-1th frame; the drone position is the center coordinate of the drone target detection frame; The drone position is concatenated with an estimated velocity of the drone to construct a first frame drone motion state vector : ； wherein is the state dimension; upper right corner denotes the transpose.

6. The method of claim 1, wherein, The step S4 specifically comprises the following steps: S41, taking the audio category probability vector the maximum probability component in the middle as the audio overall confidence of the current frame ; S42, take the visual category probability vector the maximum probability component in the middle as the visual overall confidence of the current frame ; S43, audio overall confidence with visual overall confidence linear normalization to audio fusion weight with visual fusion weight ; S44, according to the audio fusion weight with the visual fusion weight to the audio class probability vector and the visual class probability vector weighted sum to obtain the fused class probability vector : ； Wherein, C represents the total number of categories; S45, splicing the visual features, the weighted audio features, and the fused unmanned aerial vehicle category probability into a multimodal fusion feature vector : ； wherein, is a visual feature; is an audio feature; is an audio fusion weight; is a fused drone class probability; denotes a vector concatenation operation by dimension; denotes a fused feature vector is a total dimension of the fused feature vector is an audio feature dimension; is a visual feature dimension; S46, transforming the multi-modal fusion feature vector by a linear transformation projecting to a unified latent space to obtain a single-frame encoding latent vector ; S47, encode single-frame hidden vectors of the entire time sequence obtain an encoder input sequence matrix after stacking input a multi-layer self-attention encoder to obtain an overall encoding sequence containing front and rear time sequence context information .

7. The method of claim 1, wherein, The step S5 specifically comprises the following steps: S51, mapping the last time UAV motion state vector to the same hidden space as the multi-layer self-attention encoder to obtain the last time prediction state ; S52, obtain the embedding vector of the predicted state at the last time by linear transformation projecting to a latent space to obtain the embedding vector of the predicted state at the last time ; S53, embedding vector of the predicted state of the previous time with single-frame encoded hidden vector linearly transformed and added to obtain the decoding input vector of the current frame ; S54, utilizing a multi-layer self-attention decoder on the decoding input vector history sequence with the overall encoded sequence to model, obtaining a trajectory latent state representation at the current time instant ; S55, representing the trajectory latent state by a linear mapping to a predicted trajectory state vector and trained by a trajectory tracking loss function; the predicted trajectory state vector The expression is as follows: ； wherein, is the frame predicted drone position; are the frame predicted drone velocity in horizontal and vertical directions, respectively; is the output layer weight matrix; is the output layer bias vector; is the state dimension; top right corner denotes transpose; The trajectory tracking loss function adopts mean square error to supervise the whole section trajectory, and its expression is as follows: ； wherein, is a trajectory tracking loss; is the Euclidean norm, which is used to measure the distance between the real state and the predicted state; is the state vector of the UAV at the frame; S56, output the predicted trajectory state vector sequence as a sequence of drone trajectories.

8. The method of claim 4, wherein, The pre-trained acoustic feature encoding network in step S22 is a convolution-cyclic hybrid network composed of a plurality of convolution layers and cyclic layers, and the input acoustic feature , and the output is an audio feature , which is obtained by pre-training.

9. The method of claim 5, wherein, The pre-trained visual feature extraction network in step S31 is an improved YOLO backbone network, which includes convolution, feature pyramid, and backbone structure, and inputs a video image frame , and outputs a visual feature , which is obtained by pre-training.

10. The UAV identification and tracking system based on multi-modal fusion and trajectory modeling according to any one of claims 1-9, wherein, It comprises: The collection module is used for collecting continuous audio stream and continuous video stream of the key area of the airport, and obtaining an audio frame sequence after aligning time of the continuous audio stream and the continuous video stream with the image frame sequence ; wherein T is the total frame number An audio category probability extraction module is configured to extract an audio category probability vector from the audio frame sequence after the standard acoustic pre-processing, the acoustic feature encoding, the audio classification head, the category scoring, and the normalization ; The unmanned aerial vehicle motion state extraction module is configured to extract the motion state of the unmanned aerial vehicle from the image frame sequence The visual classification head is configured to input the pre-trained visual feature extraction network to perform category scoring and normalization to obtain a visual category probability vector Then, the unmanned aerial vehicle position is obtained according to the detection head, and an estimated speed of the unmanned aerial vehicle is calculated according to the positions of adjacent frames of the unmanned aerial vehicle to construct an unmanned aerial vehicle motion state vector ; The overall encoding sequence construction module is used to construct the sequence based on the audio category probability vector. With visual category probability vector The confidence level is adaptively weighted, and class fusion is performed at the probability level to obtain the fused class probability vector. And based on the fused class probability vector Construct a unified multimodal fusion feature vector at the feature level. Then, the multimodal fusion feature vectors are used. Projecting onto a unified latent space and passing it through a multi-layer self-attention encoder, we obtain a holistic encoded sequence containing preceding and following temporal context information. ; The UAV trajectory prediction output module is configured to map the UAV motion state vector at the previous time point and linearly transform the embedding vector and the single-frame encoding hidden vector to obtain a current frame decoding input vector , according to the current frame decoding input vector , the UAV trajectory prediction sequence is output after linear mapping and training of the current trajectory hidden state representation modeled by the multi-layer self-attention decoder .

Images