A target recognition method and device based on multimodal data fusion

Abstract

The present invention relates to a target recognition method and device for multimodal data fusion, comprising: inputting the multimodal data of the target to be identified into a trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network and a JEM feature extraction network that have been feature aligned; and obtaining the recognition result of the target to be identified. Through the above technical solution, the influence of missing modal data on target recognition is taken into account, and the feature extraction network is feature aligned, which improves the rationality and effectiveness of the fusion, and takes into account the influence of missing modal data on target recognition in real scenarios. Compared with traditional methods, it has more potential, better generalization performance, and better stability.

Description

Technical Field

[0001] The present invention relates to the technical field of radar target recognition, and in particular to a target recognition method and device based on multimodal data fusion. Background Art

[0002] Radar target recognition is a further development of radar target detection and tracking. To accurately identify non-cooperative targets, radar target recognition must fully utilize all available modal data. However, in real-world scenarios, due to inconsistent representations and significant differences in distribution across modal data, effectively integrating and fully utilizing these data features presents significant challenges. Consequently, target recognition faces a significant challenge in utilizing multimodal data.

[0003] However, because the various modal data acquired by radar are heterogeneous, they suffer from semantic bias and modal imbalance. Existing methods face challenges in feature extraction and modal fusion, such as semantic gaps between features, feature ambiguity, and poor modal generalization. Consequently, in multimodal data recognition scenarios, the recognition performance of existing methods can degrade dramatically or even fail. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a target recognition method and device for multimodal data fusion.

[0005] According to a first aspect of an embodiment of the present invention, a method for target recognition using multimodal data fusion is provided, the method comprising:

[0006] Inputting multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;

[0007] Obtaining a recognition result of the target to be recognized.

[0008] Optionally, the track feature extraction network includes a first input layer, a Patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer, and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer, and a second output layer connected in sequence.

[0009] Optionally, the training process of the feature extraction network includes:

[0010] Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets;

[0011] Performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal dataset to obtain a loss function;

[0012] The parameters of the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network are iteratively updated using the back propagation algorithm and the loss function until preset conditions are met, thereby obtaining the trained feature extraction network.

[0013] Optionally, the performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network based on contrastive learning and the multimodal dataset to obtain a loss function includes:

[0014] Inputting the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features;

[0015] The HRRP features and JEM features that match the track features are used as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are used as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modalities, and the negative sample pairs represent the representation of different targets in different modalities;

[0016] Calculating the similarity between the positive sample pair and the negative sample pair to obtain a similarity matrix;

[0017] The loss function is obtained according to the similarity matrix.

[0018] Optionally, the loss function is expressed as follows:

[0019]

[0020] Among them, loss represents the loss function, loss 12 Represents the first sub-loss function between track features and HRRP features, loss 13 represents the second sub-loss function between the track feature and the JEM feature; the track feature is the feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is the feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is the feature obtained by inputting the JEM into the JEM feature extraction network;

[0021] The first sub-loss function is expressed as follows:

[0022]

[0023] Among them, crossentropyloss(·) represents the loss calculation of cross entropy, S 12 is the similarity matrix between the track feature and the HRRP feature, labels is the label, axis=0 means aligning the track feature with the HRRP feature, axis=1 means aligning the HRRP feature with the track feature;

[0024] The second sub-loss function is expressed as follows:

[0025]

[0026] Among them, S 13 is the similarity matrix between the track feature and the JEM feature, axis'=0 indicates that the track feature is aligned with the JEM feature, and axis'=1 indicates that the JEM feature is aligned with the track feature.

[0027] Optionally, the feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.

[0028] Optionally, the keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer, and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features.

[0029] Optionally, the classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result based on the aggregated features.

[0030] According to a second aspect of an embodiment of the present invention, a device for target recognition using multimodal data fusion is provided, the device comprising:

[0031] An input module is configured to input multimodal data of a target to be identified into a trained recognition network; wherein the recognition network comprises a feature extraction network, a feature fusion enhancement network, a keyless attention network, and a classification network connected in sequence; the multimodal data comprises track information, HRRP echo signals, and JEM; the feature extraction network comprises a track feature extraction network, an HRRP feature extraction network, and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is configured to extract features from the track information to obtain track features; the HRRP feature extraction network is configured to extract features from the HRRP echo signals to obtain HRRP features; and the JEM feature extraction network is configured to extract features from the JEM to obtain JEM features.

[0032] The recognition module is used to obtain the recognition result of the target to be recognized.

[0033] The technical solution provided by the present invention can have the following beneficial effects:

[0034] Through the above technical solution, multimodal data is used as the basis for target recognition, and the impact of missing modal data on target recognition in real scenarios is taken into account. Compared with traditional feature fusion and decision fusion methods, feature alignment processing is performed on the feature extraction network, which fully considers the heterogeneous characteristics and semantic deviations of multimodal data, improves the rationality and effectiveness of fusion, and considers the impact of missing modal data on target recognition in real scenarios. Compared with traditional methods, it has more potential, better generalization performance, and better stability.

[0035] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:

[0037] Figure 1 The present invention is a flowchart showing an Internet threat detection and backtracking method based on adaptive abnormal behavior analysis according to an exemplary embodiment.

[0038] Figure 2The present invention is a block diagram showing a multimodal data fusion target recognition device according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] In order to facilitate the understanding of the present invention, the related art and the inventive concept of the present invention are briefly described first.

[0040] Radar target recognition uses the radar echo signal to determine the target type. Compared to traditional target detection tasks, target recognition requires measuring more in-depth, detailed target features. To achieve this, features can be obtained from multiple modalities, including radar high-resolution range profiles, modulation spectra, and track information. The radar high-resolution range profile (HRRP) is the projection vector of the target scattering point echo along the radar ray, obtained using a broadband radar signal. It provides the distribution of target scattering points along the range direction and contains rich structural information such as the target's geometric dimensions. The jet engine modulation (JEM) spectrum analyzes the target's micro-motion characteristics to obtain spectral information. By leveraging the interaction between the radar signal and the target, it extracts micro-motion features such as rotation and vibration, providing a detailed distribution of the target in the frequency domain and rich information such as the target's motion pattern and structural characteristics. Track information is the trajectory data obtained by the radar system continuously tracking the target's position changes. It records the target's motion path in space and provides dynamic information such as the target's velocity, acceleration, and direction. Track information contains the target's motion pattern and behavioral characteristics, and is of great value for target recognition and prediction tasks. At present, the study of separate recognition methods based on radar HRRP or JEM is one of the important ways to achieve radar target recognition.

[0041] Most existing target recognition methods are based on single-modal data, which may not be robust enough in practical applications. Single-modal data is easily interfered with and restricted in complex environments, resulting in reduced recognition performance. In contrast, multimodal fusion methods can provide more comprehensive target feature information by combining data from different modalities. Although single-modal data have their own advantages and disadvantages in the recognition process, they are highly complementary. The comprehensive use of HRRP, JEM and track information for recognition can provide more comprehensive and rich target features, and improve the robustness and accuracy of the recognition system. The fusion of multimodal data not only increases the amount of information, but also effectively compensates for the shortcomings of a single modality, thereby achieving more reliable target recognition in complex environments.

[0042] The prior art proposes a fusion method that utilizes both wideband and narrowband features of aircraft targets for identification. This method analyzes the characteristics of wideband and narrowband recognition systems using the power spectrum of HRRP and target speed and altitude as identification features, and utilizes DS evidence theory to achieve decision-level fusion target recognition based on wideband and narrowband recognition results. However, this method utilizes wideband and narrowband recognition results for decision fusion, without considering the semantic bias and modal imbalance of multimodal data. In actual application scenarios, when modalities are missing, the generalization performance of the model is easily affected. Furthermore, the fusion of manually selected track information features relies heavily on prior information from expert knowledge, resulting in poor adaptability.

[0043] Therefore, the present invention proposes a target recognition method based on multimodal data fusion to solve this technical problem.

[0044] Figure 1 FIG. 1 is a flow chart showing an Internet threat detection and backtracking method based on adaptive abnormal behavior analysis according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps:

[0045] S101. Inputting multimodal data of the target to be identified into a trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network, and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signals, and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network, and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signals to obtain HRRP features; and the JEM feature extraction network is used to extract features from the JEM to obtain JEM features;

[0046] S102: Obtain the recognition result of the target to be recognized.

[0047] Optionally, the feature extraction network includes a track feature extraction network, a HRRP feature extraction network and a JEM feature extraction network; the track feature extraction network includes a first input layer, a Patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer and a second output layer connected in sequence.

[0048] It can be understood that the functions of each layer of the track feature extraction network are as follows: the first input layer: the input is the track information in the multimodal data of a certain category of targets (i.e., a multidimensional track data sequence); the patch segmentation layer: the input track information sequence is divided according to the window size and step size to form multiple patch blocks; the first position encoding layer: each patch block is position-encoded using a trainable position encoding matrix Wpos; the first Transformer encoder layer: contains 12 layers of Transformer encoders, each encoder contains an 8-head attention mechanism and a feedforward neural network, and the patch block is mapped to the Transformer input potential space through a trainable linear parameter matrix Wp, with a dimension of 256; the first flattening layer: flattens the output of the Transformer encoder; the first fully connected layer: contains 3 fully connected layers, the input dimension is equal to the output dimension of the first flattening layer, and the output dimension is 128.

[0049] For example, the parameters of each layer of the HRRP feature extraction network are set as follows: First convolutional layer: 64 convolution kernels, 7×1 kernel size, 2 kernel shifting step, and 3 padding. First pooling layer: max pooling, 3×1 kernel size, 2 kernel shifting step, and 1 padding. First to fourth residual blocks: Each residual block contains two convolutional layers, with 3×1 kernel size, 1 kernel shifting step, and 1 padding. The number of convolution kernels in each residual block is 64, 128, 256, and 512, respectively. Each convolution layer is followed by a batch normalization layer and a ReLU activation function. Second fully connected layer: 512 neurons. First output layer: 128 neurons, using a linear activation function. The parameter settings of each layer of the JEM feature extraction network can be referenced for the HRRP feature extraction network and are not detailed here.

[0050] Optionally, the feature extraction network training process includes:

[0051] Construct a multimodal dataset; wherein the multimodal dataset includes track information, HRRP echo signals and JEM of multiple categories of targets;

[0052] Based on contrastive learning and multimodal datasets, the track feature extraction network, HRRP feature extraction network, and JEM feature extraction network are aligned to obtain the loss function.

[0053] Using the back-propagation algorithm and loss function, the parameters of the track feature extraction network, HRRP feature extraction network, and JEM feature extraction network are iteratively updated until the preset conditions are met, and a trained feature extraction network is obtained.

[0054] For example, a dataset can be constructed by organizing track information, HRRP echo signals, and JEMs for N target categories. The JEM is the JEM modulation spectrum of the current HRRP echo signal with the same batch and minimal time difference, based on the HRRP echo signal's time. The track information is M historical multidimensional track data points from the same batch and minimal time difference as the current HRRP echo signal. Each category contains at least 1000 radar echo signals, where N ≥ 5 and M ≥ 15.

[0055] It can be understood that through contrastive learning, the track information is used as a link to make the matching features closer in the embedding space, while the non-matching features are farther away.

[0056] Optionally, feature alignment is performed on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network based on contrastive learning and multimodal datasets to obtain a loss function, including:

[0057] The multimodal dataset is input into the track feature extraction network, HRRP feature extraction network and JEM feature extraction network to obtain track features, HRRP features and JEM features;

[0058] The HRRP features and JEM features that match the track features are used as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are used as negative sample pairs. The positive sample pairs represent the representation of the same target in different modalities, and the negative sample pairs represent the representation of different targets in different modalities.

[0059] Calculate the similarity between the positive sample pairs and the negative sample pairs to obtain the similarity matrix;

[0060] The loss function is obtained according to the similarity matrix.

[0061] It is understandable that a batch of data, including n samples, can be obtained from the multimodal data set, and the multimodal data of each sample can be input into the corresponding extraction network, that is, the data of the three modalities of each sample (track data, HRRP, JEM) are extracted using the track feature extraction network, HRRP feature extraction network and JEM feature extraction network respectively. For each track feature, the HRRP feature and JEM feature matching it are constructed as positive sample pairs. in, represents the track characteristics of the i-th sample, F i 2 represents the HRRP feature of the i-th sample, Represent the JEM features of the i-th sample; construct the HRRP features and JEM features that do not match it as a negative sample pair. A negative sample pair refers to the representation of different samples in different modalities. For example, the track features of the i-th sample and the HRRP features of the j-th sample (i≠j) constitute a negative sample pair.

[0062] The cosine similarity can be used to calculate the similarity between positive sample pairs and negative sample pairs. For all samples in a batch of data, the similarity matrix S between the track features and the HRRP features is calculated. 12 and the similarity matrix S between track features and JEM features 13 . The element s of the similarity matrix ij represents the cosine similarity between the track feature of the i-th sample and the HRRP feature or JEM feature of the j-th sample, for example, The elements on the diagonal of the similarity matrix are defined as positive sample pairs, and the rest are negative sample pairs.

[0063] Optionally, the loss function is expressed as follows:

[0064]

[0065] Among them, loss represents the loss function, loss 12 Represents the first sub-loss function between track features and HRRP features, loss 13 represents the second sub-loss function between the track feature and the JEM feature; the track feature is the feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is the feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is the feature obtained by inputting the JEM into the JEM feature extraction network;

[0066] The first sub-loss function is expressed as follows:

[0067]

[0068] Among them, crossentropyloss(·) represents the loss calculation of cross entropy, S 12 is the similarity matrix between track features and HRRP features, labels is the label, indicating the correct match of each track feature and HRRP feature pair, axis = 0 means aligning track features with HRRP features, and axis = 1 means aligning HRRP features with track features. By maximizing the similarity of correct track-HRRP pairs and minimizing the similarity of incorrect pairs, contrastive learning can learn the association between modal features in an unsupervised manner.

[0069] The second sub-loss function is expressed as follows:

[0070]

[0071] Among them, S 13 is the similarity matrix between track features and JEM features. axis' = 0 means aligning track features with JEM features. labels is the label, indicating the correct match of each track feature and JEM feature pair. axis' = 1 means aligning JEM features with track features. By maximizing the similarity of correct track-JEM pairs and minimizing the similarity of incorrect pairs, contrastive learning can learn the association between modal features in an unsupervised manner.

[0072] Optionally, the feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.

[0073] It can be understood that after completing the alignment of modal features, the attention mechanism is applied to fuse and enhance the features of each modality.

[0074] For example, the parameter settings of each layer of the feature fusion enhancement network can be referred to as follows: the fourth input layer: the input is three modal features, and the dimension of each feature vector is d; the embedding layer: maps the input modal features to a high-dimensional space, and the embedding dimension is d model Can be set to 512; Second position encoding layer: position encode each input feature, using a trainable position encoding matrix W pos , the position encoding dimension is the same as the embedding dimension, that is, d model ; The second Transformer encoder layer: contains 6 layers of Transformer encoders, each encoder contains an 8-head attention mechanism and a feedforward neural network. Transformer encoder layer parameters: the number of encoder layers is set to 6, the number of heads of the multi-head self-attention mechanism is set to 8, the hidden layer dimension of the feedforward neural network is set to 2048, and the input features are mapped to the Transformer input potential space through a trainable linear parameter matrix with a dimension of d model ; The second flattening layer: flattens the output of the Transformer encoder; The fourth fully connected layer: contains 3 fully connected layers, the input dimension is equal to the output dimension of the second flattening layer, the output dimension is 128, the number of neurons in the first layer is 512, the number of neurons in the second layer is 256, and the number of neurons in the third layer is 128. The activation function uses the ReLU activation function.

[0075] Optionally, the keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer, and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features.

[0076] It is understandable that after obtaining the fused enhanced features, a keyless attention network is used to aggregate them to effectively handle modality loss. Multiple fused enhanced features are input and an aggregated feature is output. When modality loss exists, this network only performs a weighted fusion of the existing enhanced features to obtain the aggregated feature used for classification.

[0077] For example, the parameters of each layer of the keyless attention network can be set as follows: the fifth input layer: input fusion enhanced features; weight calculation layer: calculate the weight of each input feature vector. Weight calculation formula: w i =softmax(W·T i ′), where W is the trainable weight matrix, T i The fused and enhanced features are input. The subscript indicates the vector transposition. Here, the row vector is transposed to the column vector for calculation. Weighted summation layer: All input feature vectors are weighted summed according to the calculated weights to obtain the final aggregated feature representation. Weighted summation formula: The third output layer: outputs the aggregated feature representation.

[0078] Optionally, the classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result based on the aggregated features.

[0079] For example, the parameters for each layer of the classification network can be set as follows: the sixth input layer: inputs the aggregated feature representation; the hidden layer: contains two fully connected layers, each followed by a ReLU activation function, with 25 neurons in the first hidden layer and 128 neurons in the second hidden layer; the fourth output layer: outputs the classification results, with the number of neurons equal to the number of target categories. The output layer uses the Softmax activation function to calculate the probability of each category.

[0080] It is worth mentioning that in addition to the feature extraction network, the feature fusion enhancement network, keyless attention network and classification network also need to be trained. The training process can be referred to as follows: input the aligned modal features, and obtain the prediction results through the feature fusion enhancement network, keyless attention network and classification network. The cross-entropy loss function is used to calculate the loss value between the predicted label and the true category label, and the back-propagation algorithm is used to iteratively update the parameters of the feature fusion enhancement network, keyless attention network and classification network.

[0081] Through the above technical solution, multimodal data is used as the basis for target recognition, and the impact of missing modal data on target recognition in real scenarios is taken into account. Compared with traditional feature fusion and decision fusion methods, feature alignment processing is performed on the feature extraction network, which fully considers the heterogeneous characteristics and semantic deviations of multimodal data, improves the rationality and effectiveness of fusion, and considers the impact of missing modal data on target recognition in real scenarios. Compared with traditional methods, it has more potential, better generalization performance, and better stability.

[0082] Figure 2 FIG1 is a block diagram of a target recognition device for multimodal data fusion according to an exemplary embodiment. Figure 2 The multimodal data fusion target recognition device 200 may include:

[0083] Input module 201 is used to input multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network, and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signals, and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network, and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signals to obtain HRRP features; and the JEM feature extraction network is used to extract features from the JEM to obtain JEM features.

[0084] The recognition module 202 is used to obtain a recognition result of the target to be recognized.

[0085] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0086] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the scope of protection of the present invention.

[0087] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0088] In addition, the various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A target recognition method for multimodal data fusion, characterized in that: The method comprises: Input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features; the keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features; Obtaining an identification result of the target to be identified; The training process of the feature extraction network includes: Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets; Performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal dataset to obtain a loss function; Using the back propagation algorithm and the loss function, iteratively updating the parameters of the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network until a preset condition is met, thereby obtaining the trained feature extraction network; The step of aligning the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to the contrastive learning and the multimodal dataset to obtain a loss function includes: Inputting the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features; The HRRP features and JEM features that match the track features are used as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are used as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modalities, and the negative sample pairs represent the representation of different targets in different modalities; Calculating the similarity between the positive sample pair and the negative sample pair to obtain a similarity matrix; Obtaining the loss function according to the similarity matrix; The loss function is expressed as follows: ； in, represents the loss function, represents the first sub-loss function between track features and HRRP features, represents the second sub-loss function between the track feature and the JEM feature; the track feature is the feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is the feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is the feature obtained by inputting the JEM into the JEM feature extraction network; The first sub-loss function is expressed as follows: ； in, represents the loss calculation of cross entropy, is the similarity matrix between the track feature and the HRRP feature, For labels, Indicates aligning the track feature with the HRRP feature, Indicates aligning the HRRP feature with the track feature; The second sub-loss function is expressed as follows: ； in, is the similarity matrix between the track feature and the JEM feature, Indicates aligning the track feature with the JEM feature, Indicates aligning the JEM feature with the track feature.

2. The target recognition method of multimodal data fusion according to claim 1, characterized in that: The track feature extraction network includes a first input layer, a patch segmentation layer, a first position encoding layer, a first Transformer encoder layer, a first flattening layer and a first fully connected layer connected in sequence; the HRRP feature extraction network includes a second input layer, a first convolutional layer, a first pooling layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a second fully connected layer, and a first output layer connected in sequence; the JEM feature extraction network includes a third input layer, a second convolutional layer, a second pooling layer, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block, a third fully connected layer, and a second output layer connected in sequence.

3. The target recognition method of multimodal data fusion according to claim 1, characterized in that: The feature fusion enhancement network includes a fourth input layer, an embedding layer, a second position encoding layer, a second Transformer encoder layer, a second flattening layer and a fourth fully connected layer connected in sequence; the feature fusion enhancement network is used to fuse and enhance the features output by the feature extraction network to obtain fused and enhanced features.

4. The target recognition method of multimodal data fusion according to claim 1, characterized in that: The classification network includes a sixth input layer, a hidden layer, and a fourth output layer connected in sequence; the classification network is used to obtain a recognition result based on the aggregated features.

5. A target recognition device for multimodal data fusion, characterized in that: The device comprises: An input module is used to input the multimodal data of the target to be identified into the trained recognition network; wherein the recognition network includes a feature extraction network, a feature fusion enhancement network, a keyless attention network and a classification network connected in sequence; the multimodal data includes: track information, HRRP echo signal and JEM; the feature extraction network includes a track feature extraction network, an HRRP feature extraction network and a JEM feature extraction network that have undergone feature alignment; the track feature extraction network is used to extract features from the track information to obtain track features; the HRRP feature extraction network is used to extract features from the HRRP echo signal to obtain HRRP features; the JEM feature extraction network is used to extract features from the JEM to obtain JEM features; the keyless attention network includes a fifth input layer, a weight calculation layer, a weighted summation layer and a third output layer connected in sequence; the keyless attention network is used to aggregate the fused and enhanced features to obtain aggregated features; An identification module, configured to obtain an identification result of the target to be identified; The training process of the feature extraction network includes: Constructing a multimodal data set; wherein the multimodal data set includes track information, HRRP echo signals and JEM of multiple categories of targets; Performing feature alignment on the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to contrastive learning and the multimodal dataset to obtain a loss function; Using the back propagation algorithm and the loss function, iteratively updating the parameters of the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network until a preset condition is met, thereby obtaining the trained feature extraction network; The step of aligning the track feature extraction network, the HRRP feature extraction network, and the JEM feature extraction network according to the contrastive learning and the multimodal dataset to obtain a loss function includes: Inputting the multimodal data set into the track feature extraction network, the HRRP feature extraction network and the JEM feature extraction network to obtain track features, HRRP features and JEM features; The HRRP features and JEM features that match the track features are used as positive sample pairs, and the HRRP features and JEM features that do not match each track feature are used as negative sample pairs; wherein the positive sample pairs represent the representation of the same target in different modalities, and the negative sample pairs represent the representation of different targets in different modalities; Calculating the similarity between the positive sample pair and the negative sample pair to obtain a similarity matrix; Obtaining the loss function according to the similarity matrix; The loss function is expressed as follows: ； in, represents the loss function, represents the first sub-loss function between track features and HRRP features, represents the second sub-loss function between the track feature and the JEM feature; the track feature is the feature obtained by inputting the track information into the track feature extraction network, the HRRP feature is the feature obtained by inputting the HRRP echo signal into the HRRP feature extraction network, and the JEM feature is the feature obtained by inputting the JEM into the JEM feature extraction network; The first sub-loss function is expressed as follows: ； in, represents the loss calculation of cross entropy, is the similarity matrix between the track feature and the HRRP feature, For labels, Indicates aligning the track feature with the HRRP feature, Indicates aligning the HRRP feature with the track feature; The second sub-loss function is expressed as follows: ； in, is the similarity matrix between the track feature and the JEM feature, Indicates aligning the track feature with the JEM feature, Indicates aligning the JEM feature with the track feature.

Images