A radar multi-modal attention feature alignment fusion recognition method

Abstract

The present application relates to a radar multi-modal attention feature alignment and fusion recognition method in the technical field of radar target recognition, which firstly carries out multi-modal feature generation preprocessing on the original echo data of FMCW radar, extracts one-dimensional high-resolution range image, two-dimensional range-Doppler spectrum and three-dimensional micro-motion time-frequency graph data; then applies the attention mechanism to the intra-modal deep feature extraction; then strengthens the representation ability of effective features through the global feature embedding and coding unit based on Transformer; then enters the cross-modal attention fusion module, realizes the precise alignment and deep adaptive fusion of three-modal features in the high-level semantic space in two stages; finally, the fusion features are input into the multi-layer perceptron for feature mapping, and the prediction result of the target category is output. The present application integrates radar echo features of different dimensions, solves the problem of one-sidedness of traditional single-modal recognition information, and improves the accuracy and robustness of target recognition.

Description

Technical Field

[0001] This invention belongs to the field of radar target recognition technology, and in particular to a radar multimodal attention feature alignment and fusion recognition method. Background Technology

[0002] Multimodal data fusion identification of airborne target radar echoes is a core technological path for overcoming the bottleneck of single-modal identification technology and adapting to complex combat scenarios in modern air defense early warning and airspace control. Traditional single-modal radar echo identification (such as one-dimensional HRRP, two-dimensional RD image, three-dimensional TF image, etc.) has significant information limitations: HRRP only reflects the radial scattering point distribution of the target, is sensitive to attitude changes and has difficulty distinguishing targets with similar shapes; RD spectrum focuses on the distance and velocity distribution, but is easily affected by clutter and lacks structural detail identification; micro-Doppler time-frequency image only captures the micro-motion characteristics of the target and cannot characterize core physical attributes such as shape and size. These defects directly lead to a significant drop in the accuracy of single-modal identification in complex electromagnetic environments, target maneuvering, and small sample scenarios, and it may even be unable to distinguish similar targets such as UAVs and civil aircraft. Multimodal data fusion can achieve a "1+1>2" effect by integrating radar echo features from different dimensions: on the one hand, it constructs a full-dimensional feature space through feature complementarity, covering core attributes such as target structure, motion, and shape, thus solving the problem of one-sided information in single-modal mode; on the other hand, it enhances robustness so that when data in one modality fails due to interference or occlusion, other modalities can fill in the gaps to avoid recognition failure, while cross-modal feature transfer enhances the generalization ability for small samples. Summary of the Invention

[0003] The purpose of this invention is to propose a radar multi-modal attention feature alignment fusion recognition method. By applying the attention mechanism to the intra-modal depth feature extraction and inter-modal feature alignment fusion stages respectively, a multi-modal attention feature alignment fusion model (Aircraft Radar Multi Domain Feature Cross Attention Fusion, AR-MDF-CAF) suitable for radar scenarios is constructed. This model achieves effective feature extraction, cross-modal feature alignment, and adaptive fusion of three different modalities of the radar echo of the flight target: one-dimensional HRRP, two-dimensional RD map, and three-dimensional TF map. This fully explores the complementary information between different modalities and improves the accuracy and robustness of flight target recognition in complex environments.

[0004] The technical solution to achieve the purpose of this invention is as follows: a radar multimodal attention feature alignment and fusion recognition method, comprising the following steps: Step 1: Construct a multimodal feature generation preprocessing workflow to process the raw IQ echo data of FMCW radar and generate one-dimensional HRRP, two-dimensional RD spectrum and three-dimensional TF map; Step 2: Design an intra-modal depth feature extraction module, including a one-dimensional range image feature encoder and a micro-motion time-frequency feature encoder, to perform independent intra-modal depth feature mining on the original radar echoes of each mode; Step 3: Design a global feature embedding and encoding unit based on Transformer, which receives basic features of each modality as tokens, suppresses irrelevant background and noise interference, and enhances the representation ability of effective features. Step 4: Design a cross-modal attention fusion module to achieve full-dimensional accurate alignment and deep fusion of the three modalities HRRP, RD, and TF in the distance and frequency dimensions in two stages; Step 5: System reasoning and target recognition. The time, space and frequency three-dimensional features after cross-modal alignment and fusion are flattened and input into a multilayer perceptron (MLP). The prediction results of the corresponding aerial target categories are output to complete the target recognition task.

[0005] Further, in the preprocessing process described in step 1: For one-dimensional HRRP, the model extracts the range distribution features of the target along the radar line of sight using a one-dimensional range image feature encoder, capturing the core structural information of the target's radial scattering points. Then, using the extracted range features as the basic token, it is input into the Transformer feature embedding encoder. Through an attention mechanism, it adaptively focuses on key range units that conform to the target's structural energy characteristics, suppressing irrelevant background and noise interference, and enhancing the representational ability of effective features. For two-dimensional RD spectrum, the model first transposes it, and then uses the range as the token to input it into the Transformer feature embedding encoder, focusing on the range units that can reflect the target's Doppler motion characteristics, and mining the correlation features between the target's velocity distribution and range information. For three-dimensional TF map, the model extracts the target's micro-motion time-frequency features unit by unit using a micro-motion time-frequency feature encoder, capturing the subtle motion information of the target's propeller rotation and wing vibration. Then, with the help of the Transformer feature embedding encoder, it focuses on the most discriminative micro-motion features in each range unit, filtering out redundant and interference information in the time-frequency domain.

[0006] Further, the intramodal deep feature mining described in step 2 specifically includes: For HRRP data, its initial dimension is Bx1xR, where B is the batch size and R is the number of distance units; firstly, multi-scale features are extracted through an N-layer multi-scale parallel one-dimensional convolution module, then the original input and the extracted multi-scale features are concatenated in the channel dimension, and the dimension becomes BxNCxR. Finally, a convolution kernel of size 1 is used to fuse the channels and transpose them to BxRxembed, where embed is the frequency embedding dimension, capturing the core structural information of the target's radial scattering points; For TF data, which is a four-dimensional tensor with a dimension of BxFxTxR, where F is the number of micro-Doppler units and T is the number of slow time units; using the distance unit as the number of channels, the time-frequency map in each distance unit channel is downsampled through multiple rounds of channel grouping convolution and pooling to obtain a feature map with a dimension of BxRxtxembed, where t is the dimensionality-reduced micro-motion time feature, and finally the dimension is adjusted to Bx(embed*t)xR to capture the target's subtle motion information.

[0007] Furthermore, in step 3, the global feature embedding and encoding based on Transformer involves a Transformer encoder comprising a position encoding module, a multi-head attention module, an additive residual module, and a feedforward neural network module. First, position information is injected into the input features through position encoding to overcome the limitation of the self-attention mechanism's lack of positional awareness. Then, self-attention calculation is performed through the multi-head attention module, as shown in Equation 1, where Q, K, and V are all obtained by mapping the features with fused position information through different linear layers. (1) In the formula, Q is the query matrix, representing the query for which matching information is currently being sought. Features; K is the key matrix, representing the index features being queried, and ┬ represents the matrix transpose; V is the value matrix, representing the actual content features being extracted; d k This represents the feature dimension of the K matrix.

[0008] The residual connection results are then normalized to make the training more stable. Next, the features are further extracted through the feedforward neural network. Finally, the output is subjected to residual connection and normalization again to obtain the output. The dimension remains unchanged throughout the process.

[0009] Furthermore, the cross-modal attention fusion module in step 4 is used for cross-modal feature alignment and fusion, specifically including a two-stage multi-head attention block: The first-stage multi-head attention block uses the HRRP feature (BxRxembed) as the Q input and the RD feature (BxRxembed) as the K and V inputs, mapping Q, K, and V to the same high dimension. Multiple attention heads calculate attention scores and weight them with the value V. Finally, the features of multiple heads are concatenated and mapped back to the original dimension of V to obtain the distance-frequency joint feature (BxRxembed); The second-stage multi-head attention block uses the transposed distance-frequency joint feature (BxembedxR) as the Q input and the reshaped time-frequency feature dimension as Bxembedx(t*R) as K and V for multi-head attention, finally obtaining the spatial, temporal, and frequency joint feature (Bxembedx(t*R)).

[0010] The beneficial effects of this invention are as follows: it performs outstandingly in flight target recognition tasks in complex airspace environments. First, it offers more comprehensive feature dimensions by integrating multimodal features covering target structure, motion, and shape, achieving a full-dimensional feature space construction that single-modal techniques cannot reach. Second, its recognition robustness is significantly improved. When data from one modality fails, other modalities can fill the gap, preventing recognition failure. At the same time, attention-based cross-modal alignment enhances the model's anti-interference and generalization capabilities. Third, it has extremely strong discrimination capabilities against low-speed, small, and similar targets such as UAVs. In the validation on the LSS-FMCWR-1.0 dataset, this model achieved a 100% recognition rate on the training set and a maximum recognition rate of 99.22% on the validation set, accurately identifying easily confused different rotorcraft UAV models. Attached Figure Description

[0011] Figure 1 This is a framework diagram of the multimodal attention feature alignment fusion model (AR-MDF-CAF) proposed in this invention.

[0012] Figure 2 This is a block diagram of the Transformer and multi-head attention mechanism architecture of the present invention.

[0013] Figure 3 This is the structure of the multi-head attention mechanism of the present invention.

[0014] Figure 4 This is a typical echo characteristic diagram of six types of UAVs in the LSS-FMCWR-1.0 dataset of this invention.

[0015] Figure 5 This is a flowchart of the multimodal data preprocessing process of the present invention.

[0016] Figure 6 This is a diagram of the preprocessed HRRP results in an embodiment of the present invention.

[0017] Figure 7 This is a diagram of the RD result after preprocessing in an embodiment of the present invention.

[0018] Figure 8 This is a TF result image after preprocessing in an embodiment of the present invention.

[0019] Figure 9 These are the simulation parameters in the embodiments of the present invention.

[0020] Figure 10 This is the model structure in an embodiment of the present invention.

[0021] Figure 11 This is a graph showing the model training and verification loss in an embodiment of the present invention.

[0022] Figure 12 This is a trend chart of the accuracy of each round of category verification in the embodiments of the present invention.

[0023] Figure 13 This is a confusion matrix diagram of the validation set of the last round of training for the model in this embodiment of the invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] refer to Figure 1 This invention relates to a radar multimodal attention feature alignment and fusion recognition method, the specific steps of which are as follows: Step 1: Construct a multimodal feature generation preprocessing workflow to process the raw IQ echo data of FMCW radar and generate a one-dimensional HRRP image, a two-dimensional RD spectrum, and a three-dimensional TF map. Step 2: Design an intra-modal depth feature extraction module, including a one-dimensional range image feature encoder and a micro-motion time-frequency feature encoder, to perform independent depth feature mining on the original radar echoes of each mode; Step 3: Design a global feature embedding and encoding unit based on Transformer, which receives basic features of each modality as tokens, suppresses irrelevant background and noise interference, and enhances the representation ability of effective features. Step 4: Design a cross-modal attention fusion module to achieve full-dimensional accurate alignment and deep fusion of the three modalities HRRP, RD, and TF in the distance and frequency dimensions in two stages; Step 5: System reasoning and target recognition. The time, space and frequency three-dimensional features after cross-modal alignment and fusion are flattened and input into a multilayer perceptron (MLP). The prediction results of the corresponding aerial target categories are output to complete the target recognition task.

[0026] In one embodiment of the present invention: In the preprocessing process described in step 1: For one-dimensional HRRP, the model extracts the distance distribution features of the target along the radar line of sight through a one-dimensional range image feature encoder, captures the core structural information of the target's radial scattering points, and then uses the extracted distance features as the basic token to input into the Transformer feature embedding encoder. Through an attention mechanism, it adaptively focuses on key distance units that conform to the target's structural energy characteristics, suppresses irrelevant background and noise interference, and enhances the representation ability of effective features; For two-dimensional RD spectrum, the model first transposes it, and then uses distance as the token to input into the Transformer feature embedding encoder, focusing on the distance units that can reflect the target's Doppler motion characteristics, and mining the correlation features between the target's velocity distribution and distance information; For three-dimensional TF map, the model extracts the target's micro-motion time-frequency features unit by unit through a micro-motion time-frequency feature encoder, captures the subtle motion information of the target's propeller rotation and wing vibration, and then uses the Transformer feature embedding encoder to focus on the most discriminative micro-motion features in each distance unit, filtering out redundant and interference information in the time-frequency domain.

[0027] In one embodiment of the present invention: the intramodal deep feature mining in step 2 specifically includes: for HRRP data, its initial dimension is Bx1xR, where B is the batch size and R is the number of distance units; firstly, multi-scale features are extracted through an N-layer multi-scale parallel one-dimensional convolution module, then the original input and the extracted multi-scale features are concatenated in the channel dimension, and the dimension becomes BxNCxR. Finally, a convolution kernel of size 1 is used to fuse the channels and transpose them to BxRxembed, where embed is the frequency embedding dimension, capturing the core structural information of the target's radial scattering points; for TF data, which is a four-dimensional tensor with a dimension of BxFxTxR, where F is the number of micro-Doppler units and T is the number of slow time units; using the distance unit as the number of channels, the time-frequency map in each distance unit channel is downsampled through multiple rounds of channel grouping convolution and pooling to obtain a feature map with a dimension of BxRxtxembed, where t is the micro-motion time feature after dimensionality reduction, and finally the dimension is adjusted to Bx(embed*t)xR to capture the subtle motion information of the target.

[0028] In one embodiment of the present invention: Step 3, the global feature embedding and encoding based on Transformer, involves a Transformer encoder comprising a position encoding module, a multi-head attention module, an additive residual module, and a feedforward neural network module. First, position information is injected into the input features through position encoding to overcome the limitation of the self-attention mechanism's lack of positional awareness. Then, self-attention calculation is performed through the multi-head attention module, as shown in Equation 1, where Q, K, and V are all obtained by mapping the features with fused position information through different linear layers. (1) In the formula, Q is the query matrix, representing the query features for which matching information is currently being sought; K is the key matrix, representing the index features being queried; ┬ represents the matrix transpose; and V is the value matrix, representing the actual content features being extracted. d k This represents the feature dimension of the K matrix.

[0029] The residual connection results are then normalized to make the training more stable. Next, the features are further extracted through the feedforward neural network. Finally, the output is subjected to residual connection and normalization again to obtain the output. The dimension remains unchanged throughout the process.

[0030] In one embodiment of the present invention: the cross-modal attention fusion module in step 4 is used for cross-modal feature alignment and fusion, specifically including a two-stage multi-head attention block: the first-stage multi-head attention block uses the HRRP feature (BxRxembed) as Q input and the RD feature (BxRxembed) as K and V input, maps Q, K, and V to the same high dimension, calculates the attention score of multiple attention heads and weights it with the value V, and finally concatenates the features of multiple heads and maps them back to the original dimension of V to obtain the distance-frequency joint feature of (BxRxembed); the second-stage multi-head attention block uses the transposed distance-frequency joint feature (BxembedxR) as Q input and the reshaped time-frequency feature dimension as Bxembedx(t*R) as K and V for multi-head attention, and finally obtains the spatial, temporal, and frequency joint feature of (Bxembedx(t*R)).

[0031] Example 1 This embodiment verifies the classification and recognition performance of the AR-MDF-CAF model for multiple types of rotary-wing UAVs. This embodiment uses the LSS-FMCWR-1.0: Multi-band FMCW Radar Low-Slow-Small Detection Dataset to train and validate the model. This dataset contains echo data of six types of rotary-wing UAVs collected in hovering state at a certain distance cell. Each type of UAV collects echo data of targets within a distance of 5-12 meters using Ku-band and L-band radars. The dataset contains 114 datasets. The original data format was .dat, and the dataset format was changed to .mat through data transfer. The number of rows in the data matrix represents the number of distance dimension sampling points in a single period, and the number of columns represents the number of echo sampling points. The dataset includes six types of UAVs: DJI Mavic 2 (01), DJI Phantom (02), DJI M350 (03), DJI Inspire 2 (04), DJI M600 (05), and fixed-wing UAVs (06). The typical echo characteristics of the six types of UAVs are as follows: Figure 8 As shown in the figure, distances are represented in meters. In the time-spectrum graph, time equals the product of the signal length and the modulation period. Frequency ranges from ±2 times the reciprocal of the modulation period. This dataset is rich in types, characterized by complex scenes, diverse categories, and varied tasks, and can support research on target detection, classification, and recognition.

[0032] For raw FMCW radar echo data, this embodiment constructs a unified and standardized multi-modal feature generation preprocessing workflow, such as... Figure 9 As shown, the specific implementation is as follows: First, the original IQ echo data is read, and then the positive frequency modulation half-range operation is performed to correct the frequency modulation nonlinearity and eliminate the interference of negative frequency components, laying the foundation for subsequent distance dimension processing.

[0033] In this embodiment, we used the above preprocessing procedure to process LSS-FMCWR-1.0 and create a dataset. Due to the involvement of multiple perspectives with fixed-wing aircraft, the amount of data after processing is large. For now, we only consider 5 categories of rotary-wing drones. Among them, DJI Mavic 2 (01), DJI Phantom (02), DJI M350 (03), and DJI M600 (05) each have 180 samples, and DJI Inspire 2 (04) has 135 samples. All of them are processed by dividing the data into 256 slow time lengths and 10 distance units, for a total of 855 samples. For details of each modal dimension in the samples, please refer to [link to relevant documentation]. Figure 10 The training and validation sets are split in a 0.7:0.3 ratio, with a batch size of 12. The model structure is shown below. Figure 11 The training rounds were set to 100, the learning rate was 1e-4, and the optimizer was Adam.

[0034] The model's training loss curve after 100 rounds is shown in the figure below. The training loss curve shows that the model has a strong learning ability on the training set. As the iteration progresses, the model's loss on the training set continues to decrease. Around the 40th round, the model approaches convergence, and the loss gradually approaches 0.1 or below. Subsequently, it shows a slight fluctuation and decreases, finally approaching and converging to around 0 around the 80th round.

[0035] The validation loss curve shows that it decreases rapidly in sync with the training loss in the early stages of training. Around 15 epochs, it begins to fluctuate and decrease, showing a certain gap with the training loss. At this point, the rate of decrease in training loss also decreases significantly, possibly because the model has learned some noisy features, leading to a decrease in generalization ability. However, around 65 epochs, the gap between the validation loss and the training loss gradually narrows, eventually converging to around 0.1, with a difference of less than 0.05 from the training loss, indicating that the model has good generalization ability.

[0036] The recognition accuracy of various targets on the validation set during training is as follows: Figure 12 As shown, category 0 features are the most prominent, achieving high accuracy in the early stages of training and maintaining it until the end. Furthermore, categories 2 and 3 features are also relatively easy to distinguish, both reaching over 90% accuracy after about 10 rounds. Categories 1 and 4 features are more difficult to distinguish, and the model primarily learns how to differentiate between these two categories in the later stages. Ultimately, the recognition accuracy for each category converges to 95% or higher. Statistically, the model achieves a maximum recognition rate of 100% on the training set and a maximum recognition rate of 99.22% on the validation set, indicating that the model possesses strong discriminative ability.

[0037] The confusion matrix in the last round of training is as follows Figure 13 As shown, it can be found that there is some confusion between "DJI Phantom" and "DJI Mavic 2" and "DJI M350", especially "DJI M350"; there is slight confusion between DJI "M600" and DJI "M350" and "DJI Inspire 2", which is consistent with the accuracy of various verification and identification methods.

[0038] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A radar multimodal attention feature alignment and fusion recognition method, characterized in that... By constructing a multimodal attention feature alignment and fusion model, we can achieve the extraction, cross-modal alignment, and fusion of multimodal features. The steps are as follows: Step 1: Construct a multimodal feature generation preprocessing workflow to process the raw IQ echo data of FMCW radar and generate one-dimensional HRRP, two-dimensional RD spectrum and three-dimensional TF map; Step 2: Design an intra-modal depth feature extraction module, including a one-dimensional range image feature encoder and a micro-motion time-frequency feature encoder, to perform independent intra-modal depth feature mining on the original radar echoes of each mode; Step 3: Design a global feature embedding and encoding unit based on Transformer, which receives basic features of each modality as tokens, suppresses irrelevant background and noise interference, and enhances the representation ability of effective features. Step 4: Design a cross-modal attention fusion module to achieve full-dimensional accurate alignment and deep fusion of the three modalities HRRP, RD, and TF in the distance and frequency dimensions in two stages; Step 5: System reasoning and target recognition. The time, space and frequency three-dimensional features after cross-modal alignment and fusion are flattened and input into the multilayer perceptron. The prediction results corresponding to the aerial target category are output to complete the target recognition task.

2. The radar multimodal attention feature alignment and fusion recognition method according to claim 1, characterized in that, In the preprocessing process described in step 1: For the one-dimensional HRRP, the model extracts the range distribution features of the target along the radar line of sight using a one-dimensional range image feature encoder, capturing the core structural information of the target's radial scattering points. Then, using the extracted range features as the basic token, it is input into the Transformer feature embedding encoder. Through an attention mechanism, it adaptively focuses on key range units that conform to the target's structural energy characteristics, suppressing irrelevant background and noise interference, and enhancing the representational ability of effective features. For the two-dimensional RD spectrum, the model first transposes it, and then uses the range as the token to input it into the Transformer feature embedding encoder, focusing on the range units that can reflect the target's Doppler motion characteristics, and mining the correlation features between the target's velocity distribution and range information. For the three-dimensional TF map, the model extracts the target's micro-motion time-frequency features unit by unit using a micro-motion time-frequency feature encoder, capturing the subtle motion information of the target's propeller rotation and wing vibration. Then, with the help of the Transformer feature embedding encoder, it focuses on the most discriminative micro-motion features in each range unit, filtering out redundant and interference information in the time-frequency domain.

3. The radar multimodal attention feature alignment and fusion recognition method according to claim 1, characterized in that, Step 2, the intramodal deep feature mining, specifically includes: For HRRP data, the initial dimension is Bx1xR, where B is the batch size and R is the number of distance units; firstly, multi-scale features are extracted through an N-layer multi-scale parallel one-dimensional convolution module; then, the original input and the extracted multi-scale features are concatenated in the channel dimension, resulting in a dimension of BxNCxR; finally, a convolution kernel of size 1 is used to fuse the channels and transpose them to BxRxembed, where embed is the frequency embedding dimension, capturing the core structural information of the target's radial scattering points; For TF data, which is a four-dimensional tensor with a dimension of BxFxTxR, where F is the number of micro-Doppler units and T is the number of slow time units; using distance units as the number of channels, the time-frequency map within each distance unit channel is downsampled through multiple rounds of channel-group convolution and pooling to obtain a feature map with a dimension of BxRxtxembed, where t is the dimensionality-reduced micro-motion time feature; finally, the dimension is adjusted to Bx(embed*t)xR to capture the target's subtle motion information.

4. The radar multimodal attention feature alignment and fusion recognition method according to claim 1, characterized in that, Step 3 involves global feature embedding and encoding based on Transformer. The Transformer encoder includes a position encoding module, a multi-head attention module, an additive residual module, and a feedforward neural network module. First, position information is injected into the input features through position encoding to overcome the limitation of the self-attention mechanism's lack of position awareness. Then, self-attention calculation is performed through the multi-head attention module, as shown in Equation 1. Here, Q, K, and V are all obtained by mapping the features with fused position information through different linear layers. （1） In the formula, Q is the query matrix, representing the query features for which matching information is currently being sought; K is the key matrix, representing the index features being queried; and ┬ represents the matrix transpose. V is a value matrix, representing the actual content features extracted; d k Represents the feature dimension of the K matrix; The residual connection results are then normalized to make the training more stable. Next, the features are further extracted through the feedforward neural network. Finally, the output is subjected to residual connection and normalization again to obtain the output. The dimension remains unchanged throughout the process.

5. The radar multimodal attention feature alignment and fusion recognition method according to claim 4, characterized in that, In step 4, the cross-modal attention fusion module is used for cross-modal feature alignment and fusion. Specifically, it includes a two-stage multi-head attention block: The first-stage multi-head attention block uses the HRRP feature (BxRxembed) as the Q input and the RD feature (BxRxembed) as the K and V inputs. Q, K, and V are mapped to the same high dimension. Multiple attention heads calculate attention scores and weight them with the value V. Finally, the features of multiple heads are concatenated and mapped back to the original dimension of V to obtain the distance-frequency joint feature (BxRxembed). The second-stage multi-head attention block uses the transposed distance-frequency joint feature (BxembedxR) as the Q input and the reshaped time-frequency feature dimension as Bxembedx(t*R) as K and V for multi-head attention. Finally, the spatial, temporal, and frequency joint feature (Bxembedx(t*R)) is obtained.

Images