Low-altitude multi-modal data fusion method based on attention mechanism

Abstract

The invention belongs to the technical field of deep learning, provides a low-altitude multi-modal data fusion method based on an attention mechanism, and realizes information fusion of multi-modal data of low-altitude primary radar, secondary radar, ADS-B, photoelectric video, radio detection and the like through a two-stage self-attention mechanism. According to the method, the track feature processing technology and the self-attention mechanism technology are combined, the problems of poor universality and learning performance in the prior art are solved, and the method has a heuristic effect on development of a universal multi-source data intelligent fusion framework in the fields of ground traffic management, air traffic management and the like. According to the method, various low-altitude multi-modal data input vectors with different sizes can be received and processed, and relevance, intention behaviors and the like among the vectors can be calculated, so that fusion and identification of the low-altitude multi-modal data are realized.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and proposes a low-altitude multimodal data fusion method based on an attention mechanism. Background Technology

[0002] In applications such as low-altitude general aviation and drones, multiple surveillance methods, including primary radar, secondary radar, ADS-B, electro-optical video, and radio detection, are used to continuously detect and track flying targets. At this time, it is necessary to integrate various multi-modal data such as position messages, detection intelligence, video, and images to form a unified low-altitude flight situation fusion information.

[0003] Low-altitude multimodal data is characterized by different types and spatiotemporal correlations, mainly manifested as input vectors of various sizes. Traditional neural networks cannot establish correlations between multiple related input vectors, thus failing to achieve effective fusion of low-altitude data. Furthermore, due to the presence of many vectors of varying sizes as inputs, traditional neural networks cannot fully leverage the relationships between these inputs during actual training, resulting in poor model training performance. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a low-altitude multimodal data fusion method based on an attention mechanism, comprising the following steps: Step 1: Extract feature vectors from the raw surveillance data using the track manager, image manager, and video manager. , , , ; Step 2: Enter the multimodal fusion layer and... The input feature vectors are concatenated into a matrix. and multiply by the corresponding weight matrix. , obtain the query matrix Key matrix Sum matrix The calculation formula is as follows: in, , , These are time parameter matrices for the query matrix, key matrix, and value matrix, respectively. Step 3: Utilize the obtained query matrix Bond matrix Calculate the correlation score between each input feature vector. The calculation formula is as follows: in, It is the scaling factor; Step 4: Analyze the correlation scores The matrix is ​​formed after softmax normalization. , using the obtained Sum matrix Calculate the output vector corresponding to each input feature vector. The calculation formula is as follows: ; Step 5: Perform feature dimensionality upgrade on the results obtained in Step 4.

[0005] Furthermore, step 5 includes: extracting spatial features using a gridded mapping layer and a spatial cross-attention layer, and capturing the dynamic changes of the time series using a temporal self-attention layer.

[0006] Furthermore, step 5 specifically includes: The input time series feature vectors are converted into high-dimensional matrices through a gridded mapping layer; Spatial features are extracted by performing cross-attention calculations on features in the grid using a spatial cross-attention layer. A temporal self-attention layer is used to process the feature vectors of the first n time steps to capture the dynamic temporal dependencies in the time series.

[0007] This invention combines trajectory feature processing technology and self-attention mechanism technology, which improves the problems of poor universality and learning ability in the current technology. It has an inspiring effect on the development of a universal multi-source data intelligent fusion framework for ground traffic management, air traffic management and other fields. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating the overall framework of the fusion model of the present invention. Detailed Implementation

[0009] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0010] This invention provides a low-altitude multimodal data fusion method based on an attention mechanism. It achieves information fusion of multimodal data from primary low-altitude radar, secondary radar, ADS-B, electro-optical video, and radio detection through a two-stage self-attention mechanism. The overall framework is as follows: Figure 1 As shown. This invention mainly includes the following steps: Step 1: Extract feature vectors from the raw surveillance data using the track manager, image manager, and video manager. , , , .

[0011] Step 2: Enter the multimodal fusion layer and... The input feature vectors are concatenated into a matrix. and multiply by the corresponding weight matrix. The corresponding fusion key matrix is ​​obtained. , , These are referred to as the query, key, and value matrices, respectively, and are calculated using the following formulas: in, , , These are the time parameter matrices for the query, key, and value matrices, respectively.

[0012] Step 3: Utilize the obtained and Calculate the correlation between the various input feature vectors, which is equivalent to calculating the correlation score of the multimodal attention mechanism. The calculation formula is as follows: in, It is a scaling factor used to stabilize gradient propagation and prevent numerical instability caused by excessively large dot product values.

[0013] Step 4: Analyze the correlation scores The matrix is ​​formed after softmax normalization. , using the obtained and Calculate the output vector corresponding to each input feature vector. The calculation formula is as follows: In this way, the self-attention mechanism learns the interrelationships and importance between feature vectors of different types of data (such as text, images, and videos), thereby achieving feature fusion of low-altitude targets.

[0014] Step 5: Extract spatial features using a gridded mapping layer and a spatial cross-attention layer, and use a temporal self-attention layer to capture dynamic changes in the time series, thereby improving the accuracy of intent recognition for low-altitude targets.

[0015] A gridded mapping layer transforms the input time-series feature vectors into high-dimensional matrices to capture local and global relationships between spatial features. A spatial cross-attention layer performs cross-attention calculations on the features within the grid, emphasizing the mutual influence between different spatial locations to extract more representative spatial features. Subsequently, a temporal self-attention layer processes the feature vectors from the first n time steps to capture the temporal dependencies of dynamic changes in the time series. In this way, the model can effectively learn the temporal and spatial changes in the intent of low-altitude targets, ultimately achieving accurate intent recognition.

[0016] This invention proposes a low-altitude multimodal data fusion method based on an attention mechanism. Through a two-level attention mechanism, it can not only receive and process low-altitude multimodal data input vectors of various sizes, but also calculate the correlation and intention behavior between the vectors, thereby realizing the fusion and recognition of low-altitude multimodal data.

[0017] This invention extracts feature vectors from raw surveillance data through a track manager, image manager, and video manager; then, it uses a multimodal attention mechanism to learn the interrelationships and importance between feature vectors of different types of data (such as text, images, and videos) to achieve low-altitude target feature fusion; finally, it uses a spatiotemporal attention mechanism to learn and pay attention to the dynamic changes of the accumulated feature vectors and fused vectors in the time series to achieve low-altitude target intent recognition.

[0018] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A low-altitude multimodal data fusion method based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Extract feature vectors from the raw surveillance data using the track manager, image manager, and video manager. , , , ; Step 2: Enter the multimodal fusion layer and... The input feature vectors are concatenated into a matrix. and multiply by the corresponding weight matrix. , obtain the query matrix Key matrix Sum matrix The calculation formula is as follows: in, , , These are time parameter matrices for the query matrix, key matrix, and value matrix, respectively. Step 3: Utilize the obtained query matrix Bond matrix Calculate the correlation score between each input feature vector. The calculation formula is as follows: in, It is the scaling factor; Step 4: Analyze the correlation scores The matrix is ​​formed after softmax normalization. , using the obtained Sum matrix Calculate the output vector corresponding to each input feature vector. The calculation formula is as follows: ； Step 5: Perform feature dimensionality upgrade on the results obtained in Step 4.

2. The low-altitude multimodal data fusion method based on an attention mechanism according to claim 1, characterized in that, Step 5 includes: extracting spatial features using a gridded mapping layer and a spatial cross-attention layer, and capturing the dynamic changes of the time series using a temporal self-attention layer.

3. The low-altitude multimodal data fusion method based on an attention mechanism according to claim 2, characterized in that, Step 5 specifically includes: The input time series feature vectors are converted into high-dimensional matrices through a gridded mapping layer; Spatial features are extracted by performing cross-attention calculations on features in the grid using a spatial cross-attention layer. A temporal self-attention layer is used to process the feature vectors of the first n time steps to capture the dynamic temporal dependencies in the time series.

Images