Target identification method and device for multi-modal data fusion
By extracting and aligning the multimodal data in radar target recognition, and combining feature fusion and attention mechanisms, the problem of multimodal data utilization in the prior art is solved, improving the recognition performance and system robustness.
Patent Information
- Application Number
- CN202411568906.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2044-11-05
AI Technical Summary
The existing radar target recognition technology has semantic deviations and modal imbalances in the utilization of multimodal data, resulting in degradation or failure of recognition performance.
The target recognition method of multimodal data fusion is adopted to extract and align track information, HRRP echo signals and JEM through the feature extraction network, and combine feature fusion enhancement network, key-value attention network and classification network to achieve effective fusion and recognition of multimodal data.
It improves the recognition performance of multimodal data, enhances the utilization ability of heterogeneous data, and improves the robustness and generalization performance of the identification system.
Smart Images

Figure CN119577669A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of radar target recognition technology, and in particular to a target recognition method and device for multi-modal data fusion. Background Art
[0002] Radar target recognition is a further development of radar target detection and tracking. In order to accurately identify non-cooperative targets, radar target recognition should make full use of all types of modal data. However, in real scenarios, due to the inconsistent representation and large distribution differences of various modal data, it is difficult to effectively integrate and fully utilize these data features. Therefore, the target recognition task faces serious difficulties in the use of multimodal data.
[0003] However, since the various modal data acquired by radar are heterogeneous information, there are semantic biases and modal imbalances. When performing feature extraction and modal fusion, existing methods face problems such as semantic gaps between features, feature ambiguity, and poor modal generalization capabilities. Therefore, in the multimodal data recognition scenario, the recognition performance of existing methods may degrade sharply or even fail. Summary of the invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a target recognition method and device for multi-modal data fusion.
[0005] According to a first aspect of an embodiment of the present invention, a method for target recognition based on multimodal data fusion is provided, the method comprising:
[0006] Input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;
[0007] Obtaining a recognition result of the target to be recognized.
[0008] Optionally, the track feature extraction network includes a first input layer, a Patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer, and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer, and a second output layer connected in sequence.
[0009] Optionally, the training process of the feature extraction network includes:
[0010] Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets;
[0011] Performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal data set to obtain a loss function;
[0012] The back propagation algorithm and the loss function are used to iteratively update the parameters of the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network until the preset conditions are met, thereby obtaining the trained feature extraction network.
[0013] Optionally, the performing feature alignment on the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network according to the contrastive learning and the multimodal data set to obtain a loss function includes:
[0014] Inputting the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features;
[0015] The HRRP features and JEM features that match the track features are taken as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are taken as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modes, and the negative sample pairs represent the representation of different targets in different modes;
[0016] Calculating the similarity between the positive sample pair and the negative sample pair to obtain a similarity matrix;
[0017] The loss function is obtained according to the similarity matrix.
[0018] Optionally, the loss function is expressed as follows:
[0019]
[0020] Among them, loss represents the loss function, loss 12 Represents the first sub-loss function between track features and HRRP features, loss 13 represents the second sub-loss function between the track feature and the JEM feature; the track feature is a feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is a feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is a feature obtained by inputting the JEM into the JEM feature extraction network;
[0021] The first sub-loss function is expressed as follows:
[0022]
[0023] Among them, crossentropyloss(·) represents the loss calculation of cross entropy, S 12 is the similarity matrix between the track feature and the HRRP feature, labels is the label, axis=0 means aligning the track feature with the HRRP feature, axis=1 means aligning the HRRP feature with the track feature;
[0024] The second sub-loss function is expressed as follows:
[0025]
[0026] Among them, S 13 is the similarity matrix between the track feature and the JEM feature, axis'=0 means aligning the track feature with the JEM feature, and axis'=1 means aligning the JEM feature with the track feature.
[0027] Optionally, the feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.
[0028] Optionally, the key-free attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer, and a third output layer connected in sequence; the key-free attention network is used to aggregate the fused and enhanced features to obtain aggregated features.
[0029] Optionally, the classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result based on the aggregated features.
[0030] According to a second aspect of an embodiment of the present invention, a target recognition device for multimodal data fusion is provided, the device comprising:
[0031] An input module is used to input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;
[0032] The recognition module is used to obtain the recognition result of the target to be recognized.
[0033] The technical solution provided by the present invention may include the following beneficial effects:
[0034] Through the above technical scheme, multimodal data is used as the basis for target recognition, and the impact of missing modal data on target recognition in real scenarios is considered. Compared with traditional feature fusion and decision fusion methods, feature extraction network is subjected to feature alignment processing, which fully considers the heterogeneous characteristics and semantic deviation of multimodal data, improves the rationality and effectiveness of fusion, and considers the impact of missing modal data on target recognition in real scenarios. Compared with traditional methods, it has more potential, better generalization performance, and better stability.
[0035] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:
[0037] Figure 1 The present invention is a flowchart of an Internet threat detection and backtracking method based on adaptive abnormal behavior analysis according to an exemplary embodiment.
[0038] Figure 2It is a block diagram of a target recognition device for multimodal data fusion according to an exemplary embodiment. DETAILED DESCRIPTION
[0039] In order to facilitate the understanding of the scheme of the present invention, the relevant state of the prior art and the inventive concept of the present invention are first briefly described.
[0040] Radar target recognition uses the radar echo signal of the target to determine the target type. Compared with the traditional target detection task, the target recognition task needs to measure more in-depth target details. In order to achieve this goal, features can be obtained from multiple modes, including radar high-resolution range profile, modulation spectrum and track information. Among them, the radar high-resolution range profile (HRRP) is the projection vector of the target scattering point echo in the direction of the radar ray obtained by broadband radar signal, which provides the distribution of the target scattering point along the distance direction and contains rich structural information such as the target geometric size. The modulation spectrum (Jet Engine Modulation, JEM) obtains spectrum information by analyzing the micro-motion characteristics of the target, and uses the interaction between the radar signal and the target to extract the micro-motion characteristics of the target, such as rotation and vibration, and provides the distribution of the target in the frequency domain, including rich information such as the target's motion mode and structural characteristics. Track information is the trajectory data obtained by the radar system continuously tracking the change of the target position, recording the target's motion path in space, and providing dynamic information such as the target's speed, acceleration and direction. The track information contains the target's motion mode and behavior characteristics, which is of great value for target recognition and prediction tasks. At present, the study of separate recognition methods based on radar HRRP or JEM is one of the important ways to achieve radar target recognition.
[0041] Most of the existing target recognition methods are based on single-modal data, which may not be robust enough in practical applications. Single-modal data is easily interfered and restricted in complex environments, resulting in reduced recognition performance. In contrast, multimodal fusion methods can provide more comprehensive target feature information by combining data from different modalities. Although single-modal data have their own advantages and disadvantages in the recognition process, they are highly complementary. The comprehensive use of HRRP, JEM and track information for recognition can provide more comprehensive and rich target features and improve the robustness and accuracy of the recognition system. The fusion of multimodal data not only increases the amount of information, but also effectively makes up for the shortcomings of a single modality, thereby achieving more reliable target recognition in complex environments.
[0042] In the prior art, a fusion method is proposed that uses the wide-band and narrow-band features of aircraft targets for identification. The method analyzes the characteristics of the wide-band and narrow-band identification system using the power spectrum of HRRP and the target speed and altitude as identification features, and uses the DS evidence theory to achieve decision-level fusion target identification based on the wide-band and narrow-band identification results. However, this method uses the wide-band and narrow-band identification results for decision fusion, and does not consider the semantic bias and modal imbalance of multimodal data. When faced with modal loss in actual application scenarios, the generalization performance of the model is easily affected, and the manually selected track information features are heavily dependent on the prior information of expert knowledge and have poor adaptability.
[0043] Therefore, the present invention proposes a target recognition method of multimodal data fusion to solve this technical problem.
[0044] Figure 1 is a flowchart of an Internet threat detection and backtracking method based on adaptive abnormal behavior analysis according to an exemplary embodiment. Figure 1 As shown, the method comprises the following steps:
[0045] S101, inputting the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;
[0046] S102: Obtain the recognition result of the target to be recognized.
[0047] Optionally, the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network; the track feature extraction network includes a first input layer, a Patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer, and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer, and a second output layer connected in sequence.
[0048] It can be understood that the functions of each layer of the track feature extraction network are as follows: the first input layer: the input is the track information in the multimodal data of a certain category of targets (i.e., a multidimensional track data sequence); the patch segmentation layer: the input track information sequence is segmented according to the window size and step size to form multiple patch blocks; the first position encoding layer: position encoding is performed on each patch block, using a trainable position encoding matrix Wpos; the first Transformer encoder layer: contains 12 layers of Transformer encoders, each encoder contains an 8-head attention mechanism and a feedforward neural network, and the patch block is mapped to the Transformer input potential space through a trainable linear parameter matrix Wp, with a dimension of 256; the first flattening layer: the output of the Transformer encoder is flattened; the first fully connected layer: contains 3 fully connected layers, the input dimension is equal to the output dimension of the first flattening layer, and the output dimension is 128.
[0049] For example, the parameters of each layer of the HRRP feature extraction network are set as follows: the first convolution layer: the number of convolution kernels is set to 64, the convolution kernel size is set to 7×1, the convolution kernel moving step is set to 2, and the padding size is set to 3; the first pooling layer: the pooling strategy adopts the maximum pooling method, the pooling kernel size is set to 3×1, the pooling kernel moving step is set to 2, and the padding size is set to 1; the first to fourth residual blocks: each residual block contains two convolution layers, the convolution kernel size is set to 3×1, the convolution kernel moving step is set to 1, and the padding size is set to 1. The number of convolution kernels of each residual block is 64, 128, 256, and 512 respectively, and the convolution layer of each residual block is followed by a batch normalization layer and a ReLU activation function; the second fully connected layer: the number of neurons is set to 512; the first output layer: the number of neurons is 128, and a linear activation function is used. The parameter settings of each layer of the JEM feature extraction network can be set by referring to the HRRP feature extraction network, which will not be repeated here.
[0050] Optionally, the training process of the feature extraction network includes:
[0051] Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets;
[0052] Based on contrastive learning and multimodal datasets, the track feature extraction network, HRRP feature extraction network and JEM feature extraction network are aligned to obtain the loss function.
[0053] Using the back propagation algorithm and loss function, the parameters of the track feature extraction network, HRRP feature extraction network and JEM feature extraction network are iteratively updated until the preset conditions are met to obtain a trained feature extraction network.
[0054] For example, the constructed data set can be to organize the track information, HRRP echo signals and JEM of N categories of targets, where the time of the HRRP echo signal is used as the reference, the JEM is the JEM modulation spectrum of the same batch as the current HRRP echo signal and with the smallest time difference; the track information is the historical M multi-dimensional track data points of the same batch as the current HRRP echo signal and with the smallest time difference. Each category contains at least 1000 radar echo signals, where N≥5 and M≥15.
[0055] It can be understood that through contrastive learning, the track information is used as a link to make the matching features closer in the embedding space, while the unmatched features are farther away.
[0056] Optionally, feature alignment is performed on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal data set to obtain a loss function, including:
[0057] Input the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features;
[0058] The HRRP features and JEM features that match the track features are taken as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are taken as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modalities, and the negative sample pairs represent the representation of different targets in different modalities;
[0059] Calculate the similarity between the positive sample pairs and the negative sample pairs to obtain the similarity matrix;
[0060] The loss function is obtained according to the similarity matrix.
[0061] It can be understood that a batch of data can be obtained from the multimodal data set, including n samples, and the multimodal data of each sample is input into the corresponding extraction network, that is, the data of the three modes of each sample (track data, HRRP, JEM) is extracted using the track feature extraction network, HRRP feature extraction network and JEM feature extraction network respectively. For each track feature, the HRRP feature and JEM feature matching it are constructed as positive sample pairs. in, represents the track characteristics of the i-th sample, F i 2 represents the HRRP feature of the i-th sample, Represent the JEM feature of the i-th sample; construct the HRRP feature and JEM feature that do not match it as a negative sample pair. A negative sample pair refers to the representation of different samples in different modalities. For example, the track feature of the i-th sample and the HRRP feature of the j-th sample (i≠j) constitute a negative sample pair.
[0062] The cosine similarity can be used to calculate the similarity between positive sample pairs and negative sample pairs. For all samples in a batch of data, the similarity matrix S between the track features and the HRRP features is calculated. 12 and the similarity matrix S between the track features and the JEM features 13 The element s of the similarity matrix ij represents the cosine similarity between the track feature of the i-th sample and the HRRP feature or JEM feature of the j-th sample, for example, The elements on the diagonal of the similarity matrix are defined as positive sample pairs, and the rest are negative sample pairs.
[0063] Optionally, the loss function is expressed as follows:
[0064]
[0065] Among them, loss represents the loss function, loss 12 Represents the first sub-loss function between track features and HRRP features, loss 13 represents the second sub-loss function between the track feature and the JEM feature; the track feature is the feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is the feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is the feature obtained by inputting the JEM into the JEM feature extraction network;
[0066] The first sub-loss function is expressed as follows:
[0067]
[0068] Among them, crossentropyloss(·) represents the loss calculation of cross entropy, S 12 is the similarity matrix between track features and HRRP features, labels is the label, indicating the correct match of each track feature and HRRP feature pair, axis=0 means aligning the track features with the HRRP features, and axis=1 means aligning the HRRP features with the track features; by maximizing the similarity of the correct track-HRRP pairs and minimizing the similarity of the wrong pairs, contrastive learning can be used to learn the association between modal features in an unsupervised manner;
[0069] The second sub-loss function is expressed as follows:
[0070]
[0071] Among them, S 13 is the similarity matrix between track features and JEM features, axis'=0 means aligning track features with JEM features, labels is the label, indicating the correct match of each track feature and JEM feature pair, and axis'=1 means aligning JEM features with track features. By maximizing the similarity of correct track-JEM pairs and minimizing the similarity of incorrect pairs, contrastive learning can be used to learn the association between modal features in an unsupervised manner.
[0072] Optionally, the feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.
[0073] It can be understood that after completing the alignment of modal features, the attention mechanism is applied to fuse and enhance the features of each modality.
[0074] For example, the parameter settings of each layer of the feature fusion enhancement network can be referred to as follows: the fourth input layer: the input is three modal features, and the dimension of each feature vector is d; the embedding layer: maps the input modal features to a high-dimensional space, and the embedding dimension is d model Can be set to 512; Second position encoding layer: position encoding for each input feature, using a trainable position encoding matrix W pos , the position encoding dimension is the same as the embedding dimension, that is, d model ; The second Transformer encoder layer: contains 6 layers of Transformer encoders, each encoder contains an 8-head attention mechanism and a feedforward neural network. Transformer encoder layer parameters: the number of encoder layers is set to 6, the number of heads of the multi-head self-attention mechanism is set to 8, the hidden layer dimension of the feedforward neural network is set to 2048, and the input features are mapped to the Transformer input potential space through a trainable linear parameter matrix with a dimension of d model ; The second flattening layer: flattens the output of the Transformer encoder; The fourth fully connected layer: contains 3 fully connected layers, the input dimension is equal to the output dimension of the second flattening layer, the output dimension is 128, the number of neurons in the first layer is 512, the number of neurons in the second layer is 256, the number of neurons in the third layer is 128, and the activation function uses the ReLU activation function.
[0075] Optionally, the keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer, and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features.
[0076] It can be understood that after obtaining the fused enhanced features, in order to effectively deal with the missing modality, the keyless attention network is used to aggregate the fused enhanced features. Multiple fused enhanced features are input and an aggregated feature is output. When there is a missing modality, this network only performs weighted fusion on the existing enhanced features to obtain the aggregated features for classification.
[0077] For example, the parameters of each layer of the keyless attention network can be set as follows: the fifth input layer: input fusion enhanced features; weight calculation layer: calculate the weight of each input feature vector. Weight calculation formula: w i =softmax(W·T i ′), where W is the trainable weight matrix, T i The subscript indicates the vector transposition, where the row vector is transposed to a column vector for operation; Weighted summation layer: All input feature vectors are weighted summed according to the calculated weights to obtain the final aggregate feature representation. Weighted summation formula: The third output layer: outputs the aggregated feature representation.
[0078] Optionally, the classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result based on the aggregated features.
[0079] For example, the parameters of each layer of the classification network can be set as follows: the sixth input layer: the feature representation after input aggregation; hidden layer: contains two fully connected layers, each layer is followed by a ReLU activation function, the number of neurons in the first hidden layer is set to 25, and the number of neurons in the second hidden layer is set to 128; the fourth output layer: outputs the classification results, and the number of neurons is equal to the number of target categories. The output layer uses the Softmax activation function to calculate the probability of each category.
[0080] It is worth mentioning that in addition to the feature extraction network, the feature fusion enhancement network, the keyless attention network and the classification network also need to be trained. The training process can be referred to as follows: input the aligned features of each modality, and obtain the prediction results through the feature fusion enhancement network, the keyless attention network and the classification network. The cross entropy loss function is used to calculate the loss value between the predicted label and the true category label, and the back propagation algorithm is used to iteratively update the parameters of the feature fusion enhancement network, the keyless attention network and the classification network.
[0081] Through the above technical scheme, multimodal data is used as the basis for target recognition, and the impact of missing modal data on target recognition in real scenarios is considered. Compared with traditional feature fusion and decision fusion methods, feature extraction network is subjected to feature alignment processing, which fully considers the heterogeneous characteristics and semantic deviation of multimodal data, improves the rationality and effectiveness of fusion, and considers the impact of missing modal data on target recognition in real scenarios. Compared with traditional methods, it has more potential, better generalization performance, and better stability.
[0082] Figure 2 is a block diagram of a target recognition device for multimodal data fusion according to an exemplary embodiment. Figure 2 , the multimodal data fusion target recognition device 200 may include:
[0083] Input module 201, used for inputting multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;
[0084] The identification module 202 is used to obtain the identification result of the target to be identified.
[0085] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0086] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.
[0087] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.
[0088] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.
Claims
1. A target recognition method for multimodal data fusion, characterized in that: The method comprises: Input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features; Obtaining a recognition result of the target to be recognized.
2. The target recognition method of multimodal data fusion according to claim 1, characterized in that: The track feature extraction network includes a first input layer, a patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer, and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer, and a second output layer connected in sequence.
3. The target recognition method of multimodal data fusion according to claim 2 is characterized in that: The training process of the feature extraction network includes: Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets; Performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal data set to obtain a loss function; The back propagation algorithm and the loss function are used to iteratively update the parameters of the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network until the preset conditions are met, thereby obtaining the trained feature extraction network.
4. The target recognition method of multimodal data fusion according to claim 3 is characterized in that: The step of aligning the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to the contrastive learning and the multimodal data set to obtain a loss function includes: Inputting the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features; The HRRP features and JEM features that match the track features are taken as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are taken as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modes, and the negative sample pairs represent the representation of different targets in different modes; Calculating the similarity between the positive sample pair and the negative sample pair to obtain a similarity matrix; The loss function is obtained according to the similarity matrix.
5. The target recognition method of multimodal data fusion according to claim 4 is characterized in that: The loss function is expressed as follows: Among them, loss represents the loss function, loss 12 Represents the first sub-loss function between track features and HRRP features, loss 13 represents the second sub-loss function between the track feature and the JEM feature; the track feature is a feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is a feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is a feature obtained by inputting the JEM into the JEM feature extraction network; The first sub-loss function is expressed as follows: loss 12 =crossentropyloss(S 12 ,labels,axis=0) +crossentropyloss(S 12 ,labels,axis=1); Among them, crossentropyloss(·) represents the loss calculation of cross entropy, S 12 is the similarity matrix between the track feature and the HRRP feature, labels is the label, axis=0 means aligning the track feature with the HRRP feature, axis=1 means aligning the HRRP feature with the track feature; The second sub-loss function is expressed as follows: loss 13 =crossentropyloss(S 13 ,labels,axis=0) +crossentropyloss(S 13 ,labels,axis=1); Among them, S 13 is the similarity matrix between the track feature and the JEM feature, axis'=0 means aligning the track feature with the JEM feature, and axis'=1 means aligning the JEM feature with the track feature.
6. The target recognition method of multimodal data fusion according to claim 1, characterized in that: The feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.
7. The target recognition method of multimodal data fusion according to claim 6, characterized in that: The keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer, and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features.
8. The target recognition method of multimodal data fusion according to claim 7, characterized in that: The classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result according to the aggregated features.
9. A target recognition device for multimodal data fusion, characterized in that: The device comprises: An input module is used to input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network after feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features; The recognition module is used to obtain the recognition result of the target to be recognized.
Citation Information
Patent Citations
Modulation identification method based on joint multi-modal information and domain adversarial neural network
CN115392326A
Visual language object classification method based on cross-modal intensive attention
CN115392382A
Supervised multi-modal sensor fusion method, device and equipment and storage medium
CN117036881A
Multi-mode radar HRRP target identification method and system
CN117420552A
Bearing cross-domain fault diagnosis method based on multi-modal attention adaptive network
CN117972307A
Cited By
Target echo classification method and system based on multi-modal data fusion
CN120524418A
Model training method and radar high-resolution range profile feature extraction method and device
CN121640079A
Model training method, radar high-resolution range image feature extraction method and device
CN121640079B
