Sea surface target identification method based on multi-modal fusion

Abstract

The invention discloses a sea surface target identification method based on multi-modal fusion, and relates to the technical field of target identification, and the method comprises the following steps: 1, GAF angle field: GAF can convert time sequence data into image data, can maintain the complete information of a signal, and also can maintain the dependence of the signal on time; and after the signal data is converted into the image data, features can be extracted by fully utilizing the advantages of the 2D-CNN in image classification and recognition. According to the sea surface target identification method based on multi-modal fusion, the BERT after the self-attention is improved through the re-attention can effectively learn and extract the sequence features after the one-dimensional HRRP and the GAF are fused. Experimental results show that the Rf-BERT can effectively learn the time features in the HRRP sequence, and the recognition performance of the radar HRRP target is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a sea surface target recognition method based on multi-modal fusion. Background Art

[0002] The resolution of high-resolution broadband radar is much smaller than the target size. The radar target echo is also called the one-dimensional high-resolution range profile (HRRP) of the target. HRRP is the amplitude of the coherent sum of the complex time echoes of the target scatterer in each ranging unit. It represents the projection of the complex echo of the target scattering center on the radar line of sight. Compared with the traditional synthetic aperture radar (SAR) and inverse synthetic aperture radar (ISAR), HRRP has the advantages of convenient acquisition, small data volume and fast processing speed. HRRP contains rich target structure features, such as target size, scatterer distribution, etc. Therefore, radar HRRP target recognition has received widespread attention in the radar automatic target recognition (RATR) community.

[0003] As deep learning has developed rapidly with its advantages such as automatic feature extraction and powerful classification and regression capabilities, more and more researchers are paying attention to this data-driven learning method, hoping to improve the recognition performance of radar HRRP targets. CNN has been successfully applied to RATR with HRRP as the information source. In the CNN model, 1D-CNN is used to process sequences, and 2D-CNN is mainly used to process images. In addition to being used directly for target recognition after preprocessing, the one-dimensional range image can also be converted into an image for recognition. Summary of the invention

[0004] The present invention provides a sea surface target recognition method based on multi-modal fusion, so as to solve the problem of making full use of target information in HRRP sequence and improving the recognition performance of radar HRRP target.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A sea surface target recognition method based on multimodal fusion, the target recognition method comprises the following steps:

[0007] Step 1: GAF angle field: GAF can convert time series data into image data, which can not only retain the complete information of the signal, but also maintain the signal's dependence on time. After the signal data is converted into image data, the advantages of 2D-CNN in image classification and recognition can be fully utilized to extract features.

[0008] Step 2: Rf-BERT network: Use the Gram Angle Sum Field (GAFS) in GAF to convert the one-dimensional range image into a two-dimensional feature image; then use CNN-reBert and 2D-ResNet18 network modules to extract the feature information in the one-dimensional range image and the two-dimensional feature map respectively, and splice and fuse them into new feature information; finally, use the attention layer, two fully connected layers and softmax to output probability distribution to complete the target classification;

[0009] Step 3: Data and experimental analysis.

[0010] Preferably, the idea of ​​step 1 is mainly derived from the concept of Gramian matrix in linear algebra; a matrix composed of the inner product of any k (k < n) vectors in n-dimensional Euclidean space is called a Gramian matrix; the Gramian matrix can reflect the linear correlation between vectors; it can be expressed as:

[0011]

[0012] where u1, u2, ..., u k are all n-dimensional vectors; the inner product of each vector can be expressed as:

[0013]

[0014] HRRP is a combination of distance units and can be expressed as X = [x1, x2, ..., x n ], where X represents the entire HRRP data and n represents the data length; for HRRP data, the first step is to scale X to [-1, 1] or [0, 1], which can be achieved by formula (3) or (4), respectively:

[0015]

[0016] or

[0017] Scaling can prevent the inner product from being biased towards a value that is too large. The Gramian matrix of HRRP can be expressed as:

[0018]

[0019] Therefore, the HRRP time series can be recalibrated using polar coordinates Encode the value as the cosine of the angle and the timestamp as the radius; the formula is as follows:

[0020]

[0021] Among them, ti is the timestamp of the moment, r i It is a polar axis, which preserves the temporal relationship; is the polar angle, retaining the numerical relationship;

[0022] First, HRRP is scaled to [-1, 1] using formula (3), and then the Cartesian coordinate system is converted to the polar coordinate system using formula (6); finally, the time correlation within different time intervals can be identified by using the angle perspective through the trigonometric sum or difference between each point, and a Gramian-like matrix that maintains time dependence is generated; there are two methods to generate the matrix: Gramian Sum Angular Field (GASF) and Gramian Difference Angular Field (GADF), which are defined as follows:

[0023]

[0024]

[0025] Through the GAF algorithm, the one-dimensional HRRP is converted into a two-dimensional image, thereby applying computer vision technology to the study of HRRP.

[0026] Preferably, the step 2 includes the following:

[0027] A1: ResNet-18: ResNet uses a residual structure to alleviate the degradation problem in deep networks;

[0028] CNN-reBert module: In ResNet, a residual block can be expressed as:

[0029] y=F(x,{W i})+x (9)

[0030] Where x and y are the input and output vectors of the layers under consideration, and W i is the weight matrix, and F is the residual function; F can be expressed as follows:

[0031] F=W2g(W1x+b1)+b2 (10)

[0032] Where W1, W2 are weight matrices, b1 and b2 are biases, and g represents the activation function ReLU;

[0033] A2: CNN-reBert module: In the CNN-reBert module, the number of channels of the input one-dimensional HRRP is first increased to 512, and a 1×5 convolution kernel is used for one-dimensional convolution; on the one hand, the extracted sequence features better retain the local structural information in HRRP; on the other hand, the dimension and number of channels of the convolution kernel independently control the dimension and sequence length of the features, respectively, realizing the decoupling of the two and improving the whole network’s ability to represent HRRP data; in addition, the size and stride of the convolution kernel are controllable, avoiding fixed position constraints and enhancing the initial expression ability of the network; positional encoding (Positional embedding) is used to add temporal information to the one-dimensional distance image, avoiding the problem that the cyclic structure is difficult to parallelize; in CNN-reBert, a total of 4 layers of reBERT models are stacked, namely N bert =4, using the 8 multi-head attention and forward connection modules in the Bert module to effectively extract the radar HRRP features and provide better input for the classification network;

[0034] A3: Hyperparameters.

[0035] Preferably, A2 comprises the following:

[0036] A21: Self-attention mechanism: The self-attention mechanism can use matrix multiplication in parallel, which is faster and more space-efficient.

[0037] Sequence feature F after positional encoding emb Perform three transformations respectively to obtain d q The query matrix Q of dimension d k The key matrix K and d v Dimensional value matrix V:

[0038] Q=F emb W q

[0039] K=F emb W k

[0040] V=F emb W v

[0041] in d q =d k ;d model is the length of the input sequence features;

[0042] Multiply Q and K to produce an attention map, which represents the correlation between all tokens in each layer; then divide by the scaling factor Finally, the attention weight of V is obtained through the softmax function, and its formula is:

[0043]

[0044] A22: Multi-head self-attention mechanism: HRRP data has a typical multi-subspace distribution structure, and the target structure information can be obtained from multiple subspaces. The multi-head attention mechanism is used to map the input to multiple subspaces, and deeply extract information that is helpful for recognition from multiple dimensions. The matrix Q, matrix K, and matrix V are linearly projected h times, that is, they are projected into h different subspaces and calculated separately; then all the calculation results are spliced, and finally a linear transformation is performed to obtain the output F multi :

[0045] F multi =Concat(h ead1, h ead2,..., h ead h )W O (12)

[0046] Where h is the number of longs, are three sets of weight matrices, is the output projection matrix;

[0047] A23: Re-attention: Use the head's attention map as the basis and generate a set of new attention maps through dynamic aggregation; for this purpose, a learnable transformation matrix θ∈R is defined h×h , and then mix the multi-head attention map into the regenerated new map before multiplying it with V; the original self-attention and re-attention; Re-attention replaces the overly simple trade-off of dropout, and the implementation is simple, the formula is as follows:

[0048]

[0049] Among them, the transformation matrix θ is multiplied by the self-attention map along the head dimension; Norm here is a normalization function used to reduce the variance of each layer; θ can be learned end-to-end;

[0050] A24: Attention layer: The attention layer can dynamically weight the fused features. In order to ensure that the fused features useful for recognition can be used more efficiently, we introduce the attention mechanism to the feature sequence O of the fused output. bert Weighted along the feature dimension, we get feature F att :

[0051]

[0052] Among them, O bert (k, l) is the first element in the kth channel feature of the fused output feature, and α(k, l) represents the weight of the corresponding element.

[0053] Preferably, A3 comprises the following:

[0054] A31: Loss function:

[0055] The loss function is the cross entropy loss function (CrossEntropyLoss) which is currently widely used in classification tasks; CrossEntropyLoss can be directly called in Pytorch, and its calculation formula is as follows:

[0056]

[0057] The log has base e; the first parameter x is the input and the output of the last layer of the network, and its shape is [batchsize, class];

[0058] To get the final cross entropy loss in Pytorch, you also need to go through the loss weight, which is calculated as follows:

[0059] loss(x, class)=weigh t[class](-x[class]+log(∑exp(x[i]))) (16)

[0060] A32: Optimizer: Use Adam optimizer;

[0061] A33: Learning rate: Use the OneCycleLR method to adjust the learning rate, that is, increase the learning rate in the initial training process to make the model learn faster, and slowly reduce the learning rate at a certain stage of training to allow the model to find the global optimal solution.

[0062] Preferably, the step three includes the following:

[0063] B1: Data preprocessing: Preprocessed data will make it easier to find the minimum value during the iteration of the gradient descent method, which will speed up the model convergence. Preprocessing includes the following two steps:

[0064] B11: Normalize the amplitude of HRRP; the data after normalization of the nth HRRP amplitude is expressed as where max(|x n |) represents the maximum absolute value of all elements in HRRP;

[0065] B12: subtract the mean of the normalized HRRP data from the respective elements;

[0066] B2: Dataset: There are two methods to obtain target echo signals, namely, actual measurement and theoretical calculation. Since most ship targets are non-cooperative targets, it is very difficult to obtain HRRP by field measurement. Six ship models were established using 3D Max, and HRRP was calculated using FEKO. The simulation parameters set included the radar center frequency of 10 GHz, bandwidth of 80 MHz, number of frequency sampling points of 200, azimuth range of 0-360°, interval of 1°, and grazing angle of 10°. The HRRP obtained has 200 range cells, and the length of each range cell is 1.875 m.

[0067] In order to meet the sample data volume requirements during neural network training and prevent overfitting, the data set needs to be expanded; two expansion methods are selected:

[0068] B21: Data enhancement for HRRP; When calculating HRRP, the coordinate axis coincides with the center of the ship, so the effective HRRP information is generally in the middle area; In the actual environment, when the radar detects a target, the echo signal may be incomplete or partially lost; Using data enhancement technology, the HRRP signal is moved along the distance direction to expand the data set, each time 20 distance units, twice to the left and twice to the right; The deleted data is discarded, and the blank part is filled with 0 using zero-padding, that is, the blank features are filled with 0; By taking those overlapping but different HRRP samples, the number of samples is increased by 4 times;

[0069] B22: Add random noise to the translated HRRP data; add 5 times Gaussian white noise to the data, and the data after adding noise meets a certain signal-to-noise ratio;

[0070] The expanded data set is divided into a training set and a test set according to the ratio of 8:2 for each type of ship target. The samples of the training set and the test set are 43200 and 10800 respectively. Since this paper adopts the supervised learning method, in the process of data set division, the category label of the data should be constructed at the same time, and each label should correspond to each group of data one by one. In addition, in order to ensure the authenticity and effectiveness of the classification and recognition performance of the network model, the entire data set needs to be randomly shuffled and reorganized before division.

[0071] B3: Comparative experiment: SVM has the highest recognition accuracy of 94.89% for TUO, but the lowest recognition accuracy of 80.83% for LCS, and the overall weighted accuracy is 89.96%; CNN has the highest recognition accuracy of 90.11% for CVN-65, but the lowest recognition accuracy of 74.5% for YOU, and the overall weighted recognition rate is 82.43%; the weighted accuracy of TATDRNN and LSTM are close, 93.79% and 94.87% respectively, but both have a probability of more than 3% to mistake TUO for YOU, LCS and DDG104; CNN-BERT has a high overall weighted accuracy of 96.9%, and the recognition accuracy of each category is also high; and the overall weighted accuracy of Rf-BERT reaches 98%, and compared with CNN-BERT, the recognition of each category is more accurate;

[0072] B4: Ablation experiment: When only the original Transformer encoder is used for classification, the weighted accuracy is only 94.32%; but after using a simple convolution block to extract features in advance, the accuracy will increase a little to 95.28%; then after adding position encoding to the features extracted by the convolution block, the weighted accuracy will reach 95.42%; if only position encoding is added to the original Transformer encoder, the weighted accuracy can reach 94.68%; replace the ordinary multi-head self-attention with Re-attent ion, the accuracy will increase slightly to 95.53%, but it can effectively avoid attention collapse; using the pre-trained two-dimensional Resnet18 alone to classify and recognize the GAF images converted from HRRP, the accuracy can reach 97.02%; if the HRRP features extracted by the previous CNN-reBert module are added for recognition and classification, the overall weighted accuracy will be slightly improved to 97.63%; finally, with the addition of the attention layer, the complete Rf-BERT will be assembled, and the overall weighted recognition accuracy can reach 98.06%.

[0073] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0074] 1. The present invention provides a sea surface target recognition method based on multimodal fusion. The BERT after improving self-attention through re-attention can effectively learn and extract the sequence features after one-dimensional HRRP and GAF ​​fusion. Experimental results show that Rf-BERT can effectively learn the temporal features in the HRRP sequence and improve the recognition performance of radar HRRP targets.

[0075] 2. The present invention provides a sea surface target recognition method based on multimodal fusion. The radar target object can be represented by a scattering point model, so HRRP includes the distribution of the target in the direction of the radar line of sight; however, the target often produces a relative displacement in the direction of the radar line of sight, and the corresponding echo will also change relatively, so it is necessary to preprocess the data; through the preprocessed data, it will be easier to find the minimum value in the iterative process of the gradient descent method, so that the model converges faster.

[0076] 3. The present invention provides a sea surface target recognition method based on multimodal fusion. By comparing with a variety of existing models, the experimental results show that Rf-BERT can achieve the best recognition accuracy under different signal-to-noise ratio noises. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of the process of converting HRRP into a GAF image according to the present invention;

[0078] Figure 2 Schematic diagram of the overall architecture of Rf-BERT of the present invention;

[0079] Figure 3 Schematic diagram of a residual block of the present invention;

[0080] Figure 4 Schematic diagram of (A) after convolution (B) position coding texture (C) after position coding of the present invention;

[0081] Figure 5 (Left) Original self-attention of the present invention; (Right) Schematic diagram of re-attention;

[0082] Figure 6 Schematic diagram of a learning rate curve of the present invention;

[0083] Figure 7 It is a schematic diagram of HRRP data enhancement of the present invention;

[0084] Figure 8 Schematic diagram of the loss function of the comparison model of SVM of the present invention;

[0085] Fig. 9 Schematic diagram of the loss function of the comparison model of CNN of the present invention;

[0086] Fig.10 Schematic diagram of the loss function of the comparison model of TATDRNN of the present invention;

[0087] Fig.11 Schematic diagram of the loss function of the comparison model of LSTM of the present invention;

[0088] Fig.12 Schematic diagram of the loss function of the comparison model of CNN-BERT of the present invention;

[0089] Fig.13 Schematic diagram of the comparison model loss function of Rf-BERT of the present invention. DETAILED DESCRIPTION

[0090] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0091] like Figure 1-Figure 13 As shown, a sea surface target recognition method based on multimodal fusion includes the following steps:

[0092] Step 1: GAF angle field: GAF can convert time series data into image data, which can not only retain the complete information of the signal, but also maintain the signal's dependence on time. After the signal data is converted into image data, the advantages of 2D-CNN in image classification and recognition can be fully utilized to extract features.

[0093] Step 2: Rf-BERT network: Use the Gram Angle Sum Field (GAFS) in GAF to convert the one-dimensional range image into a two-dimensional feature image; then use CNN-reBert and 2D-ResNet18 network modules to extract the feature information in the one-dimensional range image and the two-dimensional feature map respectively, and splice and fuse them into new feature information; finally, use the attention layer, two fully connected layers and softmax to output probability distribution to complete the target classification;

[0094] Step 3: Data and experimental analysis:

[0095] The idea mainly comes from the concept of Gramian matrix in linear algebra; a matrix composed of the inner product of any k (k < n) vectors in n-dimensional Euclidean space is called a Gramian matrix; the Gramian matrix can reflect the linear correlation between vectors; it can be expressed as:

[0096]

[0097] where u1, u2, ..., u k are all n-dimensional vectors; the inner product of each vector can be expressed as:

[0098]

[0099] HRRP is a combination of distance units and can be expressed as X = [x1, x2, ..., x n], where X represents the entire HRRP data and n represents the data length; for HRRP data, the first step is to scale X to [-1, 1] or [0, 1], which can be achieved by formula (3) or (4), respectively:

[0100]

[0101] or

[0102] Scaling can prevent the inner product from being biased towards a value that is too large. The Gramian matrix of HRRP can be expressed as:

[0103]

[0104] Therefore, the HRRP time series can be recalibrated using polar coordinates Encode the value as the cosine of the angle and the timestamp as the radius; the formula is as follows:

[0105]

[0106] Among them, t i is the timestamp of the moment, r i It is a polar axis, which preserves the temporal relationship; is the polar angle, retaining the numerical relationship;

[0107] First, HRRP is scaled to [-1, 1] using formula (3), and then the Cartesian coordinate system is converted to the polar coordinate system using formula (6); finally, the time correlation within different time intervals can be identified by using the angle perspective through the trigonometric sum or difference between each point, and a Gramian-like matrix that maintains time dependence is generated; there are two methods to generate the matrix: Gramian Sum Angular Field (GASF) and Gramian Difference Angular Field (GADF), which are defined as follows:

[0108]

[0109]

[0110] Through the GAF algorithm, the one-dimensional HRRP is converted into a two-dimensional image, thereby applying computer vision technology to the study of HRRP.

[0111] The second step includes the following:

[0112] A1: ResNet--18:

[0113] CNN can learn features of different layers through multiple convolutions. As the number of layers increases, rich features of different layers are extracted by CNN. However, simply increasing the number of layers does not improve performance, because as the network deepens, the training and optimization of the network will become more and more difficult. ResNet solves the above problems and brings performance improvement. Compared with the CNN model, this architecture shows excellent performance in various image processing tasks.

[0114] ResNet uses a residual structure to alleviate the degradation problem in deep networks; CNN-reBert module:

[0115] In ResNet, a residual block can be expressed as:

[0116] y=F(x,{W i})+x (9)

[0117] Where x and y are the input and output vectors of the layers under consideration, and W i is the weight matrix, and F is the residual function; F can be expressed as follows:

[0118] F=W2g(W1x+b1)+b2 (10)

[0119] Where W1, W2 are weight matrices, b1 and b2 are biases, and g represents the activation function ReLU;

[0120] ResNet is a well-balanced CNN model that is widely used in computer vision tasks. After the HRRP sequence is converted into an image, 2D-ResNet can be used for classification training. This paper uses ResNet18 to extract features from GASF images.

[0121] A2: CNN-reBert module:

[0122] Bert is based on the transformer structure and uses a multi-head self-attention mechanism to model text. It can not only capture long-distance text features, but also pay attention to the words related to it in the context when encoding the current word, focusing on important information, and learning deep context associations through a deep network structure, enhancing the model's expressiveness. The core logic of the attention mechanism is to mine key information from global information and give it more attention;

[0123] In the CNN-reBert module, the number of channels of the input one-dimensional HRRP is first increased to 512, and a 1×5 convolution kernel is used for one-dimensional convolution. On the one hand, the extracted sequence features better retain the local structural information in HRRP. On the other hand, the dimension and number of channels of the convolution kernel independently control the dimension and sequence length of the features, respectively, realizing the decoupling of the two and improving the whole network's ability to represent HRRP data. In addition, the size and stride of the convolution kernel are controllable, avoiding fixed position constraints and enhancing the initial expression ability of the network. Positional embedding is used to add temporal information to the one-dimensional distance image, avoiding the problem that the cyclic structure is difficult to parallelize. In CNN-reBert, a total of 4 layers of reBERT models are stacked, namely N bert =4, using the 8 multi-head attention and forward connection modules in the Bert module to effectively extract the radar HRRP features and provide better input for the classification network;

[0124] A3: Hyperparameters.

[0125] The A2 includes the following:

[0126] A21: Self-attention mechanism: The self-attention mechanism can use matrix multiplication in parallel, which is faster and more space-efficient.

[0127] Sequence feature F after positional encoding emb Perform three transformations respectively to obtain d q The query matrix Q of dimension d k The key matrix K and d v Dimensional value matrix V:

[0128] Q=F emb W q

[0129] K=F emb W k

[0130] V=F emb W v

[0131] in d q =d k ;d model is the length of the input sequence features;

[0132] Multiply Q and K to produce an attention map, which represents the correlation between all tokens in each layer; then divide by the scaling factor Finally, the attention weight of V is obtained through the softmax function, and its formula is:

[0133]

[0134] A22: Multi-head self-attention mechanism: HRRP data has a typical multi-subspace distribution structure, and the target structure information can be obtained from multiple subspaces. The multi-head attention mechanism is used to map the input to multiple subspaces, and deeply extract information that is helpful for recognition from multiple dimensions. The matrix Q, matrix K, and matrix V are linearly projected h times, that is, they are projected into h different subspaces and calculated separately; then all the calculation results are spliced, and finally a linear transformation is performed to obtain the output F multi :

[0135] F multi =Concat(h ead1, h ead2,..., h ead h )W O (12)

[0136] Where h is the number of longs, are three sets of weight matrices, is the output projection matrix;

[0137] A23: Re-attention: Use the head's attention map as the basis and generate a set of new attention maps through dynamic aggregation; for this purpose, a learnable transformation matrix θ∈R is defined h×h , and then mix the multi-head attention map into the regenerated new map before multiplying it with V; the original self-attention and re-attention; Re-attention replaces the overly simple trade-off of dropout, and the implementation is simple, the formula is as follows:

[0138]

[0139] Among them, the transformation matrix θ is multiplied by the self-attention map along the head dimension; Norm here is a normalization function used to reduce the variance of each layer; θ can be learned end-to-end;

[0140] The advantages of the proposed re-attention are two-fold. First, compared to other possible attention enhancement methods (such as randomly dropping some elements of the attention map or adjusting the SoftMax temperature), re-attention exploits the interactions between different attention heads to collect their complementary information and better improves the diversity of the attention map. Compared to the original self-attention, re-attention requires only a few lines of code and negligible computational overhead. Therefore, it is more effective than methods that increase the embedding dimensionality.

[0141] A24: Attention layer: The attention layer can dynamically weight the fused features. In order to ensure that the fused features useful for recognition can be used more efficiently, we introduce the attention mechanism to the feature sequence O of the fused output. bert Weighted along the feature dimension, we get feature F att :

[0142]

[0143] Among them, O bert (k, l) is the first element in the kth channel feature of the fused output feature, and α(k, l) represents the weight of the corresponding element;

[0144] According to the importance of feature dimension elements, the network automatically learns different weights.

[0145] The A3 includes the following:

[0146] A31: Loss function:

[0147] The loss function is the cross entropy loss function (CrossEntropyLoss) which is currently widely used in classification tasks; CrossEntropyLoss can be directly called in Pytorch, and its calculation formula is as follows:

[0148]

[0149] The log has base e; the first parameter x is the input and the output of the last layer of the network, and its shape is [batchsize, class];

[0150] To get the final cross entropy loss in Pytorch, you also need to go through the loss weight, which is calculated as follows:

[0151] loss(x, class)=weigh t[class](-x[class]+log(∑exp(x[i]))) (16)

[0152] A32: Optimizer: Adam optimizer is selected; it combines the advantages of the three optimization algorithms of Monument, AdaGrad and RMSProp. By calculating the first-order moment estimation and second-order moment estimation of the gradient, it designs independent adaptive learning rates for different parameters. It has the advantages of simple implementation, efficient calculation, low memory requirement, and is suitable for unstable objective functions.

[0153] A33: Learning rate: Use the 0neCycleLR method to adjust the learning rate, that is, increase the learning rate in the initial training process to make the model learn faster, and slowly reduce the learning rate when the training reaches a certain stage to allow the model to find the global optimal solution.

[0154] The step three includes the following:

[0155] B1: Data preprocessing: Radar target objects can be represented by scattering point models, so HRRP includes the distribution of targets in the radar line of sight. However, targets often have relative displacements in the radar line of sight, and the corresponding echoes also change relatively, so data preprocessing is required. Preprocessed data will make it easier to find the minimum value during the iteration of the gradient descent method, which will speed up the model convergence. Preprocessing includes the following two steps:

[0156] B11: Normalize the amplitude of HRRP; the data after normalization of the nth HRRP amplitude is expressed as where max(|x n |) represents the maximum absolute value of all elements in HRRP;

[0157] B12: subtract the mean of the normalized HRRP data from the respective elements;

[0158] B2: Dataset: There are two methods to obtain target echo signals, namely, actual measurement and theoretical calculation. Since most ship targets are non-cooperative targets, it is very difficult to obtain HRRP by field measurement. Six ship models were established using 3D Max, and HRRP was calculated using FEKO. The simulation parameters set included the radar center frequency of 10 GHz, bandwidth of 80 MHz, number of frequency sampling points of 200, azimuth range of 0-360°, interval of 1°, and grazing angle of 10°. The HRRP obtained has 200 range cells, and the length of each range cell is 1.875 m.

[0159] In order to meet the sample data volume requirements during neural network training and prevent overfitting, the data set needs to be expanded; two expansion methods are selected:

[0160] B21: Data enhancement for HRRP; When calculating HRRP, the coordinate axis coincides with the center of the ship, so the effective HRRP information is generally in the middle area; In the actual environment, when the radar detects a target, the echo signal may be incomplete or partially lost; Using data enhancement technology, the HRRP signal is moved along the distance direction to expand the data set, each time 20 distance units, twice to the left and twice to the right; The deleted data is discarded, and the blank part is filled with 0 using zero-padding, that is, the blank features are filled with 0; By taking those overlapping but different HRRP samples, the number of samples is increased by 4 times;

[0161] B22: Add random noise to the translated HRRP data; add 5 times Gaussian white noise to the data, and the data after adding noise meets a certain signal-to-noise ratio;

[0162] The expanded data set is divided into a training set and a test set according to the ratio of 8:2 for each type of ship target. The samples of the training set and the test set are 43200 and 10800 respectively. Since this paper adopts the supervised learning method, in the process of data set division, the category label of the data should be constructed at the same time, and each label should correspond to each group of data one by one. In addition, in order to ensure the authenticity and effectiveness of the classification and recognition performance of the network model, the entire data set needs to be randomly shuffled and reorganized before division.

[0163] B3: Comparative experiment: SVM has the highest recognition accuracy of 94.89% for TUO, but the lowest recognition accuracy of 80.83% for LCS, and the overall weighted accuracy is 89.96%; CNN has the highest recognition accuracy of 90.11% for CVN-65, but the lowest recognition accuracy of 74.5% for YOU, and the overall weighted recognition rate is 82.43%; the weighted accuracy of TATDRNN and LSTM are close, 93.79% and 94.87% respectively, but both have a probability of more than 3% to mistake TUO for YOU, LCS and DDG104; CNN-BERT has a high overall weighted accuracy of 96.9%, and the recognition accuracy of each category is also high; and the overall weighted accuracy of Rf-BERT reaches 98%, and compared with CNN-BERT, the recognition of each category is more accurate;

[0164] Table 1. Comparative experiments

[0165] Model Weighted Precision Weighted Recall Weighted F1 score Weighted Accuracy SVM 89.5278 89.9570 89.4640 89.9570 CNN 82.4870 82.4259 82.3951 82.4259 TATDRNN 93.8723 93.7917 93.8146 93.7917 LSTM 94.9766 94.8704 94.8835 94.8704 CNN-BERT 96.9381 96.9074 96.9123 96.9074 Rf-BERT 98.0555 98.0619 98.0559 98.0619

[0166] B4: Ablation experiment: When only the original Transformer encoder is used for classification, the weighted accuracy is only 94.32%; but after using a simple convolution block to extract features in advance, the accuracy will increase a little to 95.28%; then after adding position encoding to the features extracted by the convolution block, the weighted accuracy will reach 95.42%; if only position encoding is added to the original Transformer encoder, the weighted accuracy can reach 94.68%; replace the ordinary multi-head self-attention with Re-attent ion, the accuracy will increase slightly to 95.53%, but it can effectively avoid attention collapse; using the pre-trained two-dimensional Resnet18 alone to classify and recognize the GAF images converted from HRRP, the accuracy can reach 97.02%; if the HRRP features extracted by the previous CNN-reBert module are added for recognition and classification, the overall weighted accuracy will be slightly improved to 97.63%; finally, with the addition of the attention layer, the complete Rf-BERT will be assembled, and the overall weighted recognition accuracy can reach 98.06%.

[0167] The working principle of the present invention is as follows: first, the Gram Angle Sum Field (GAFS) in GAF is used to convert the preprocessed one-dimensional range image into a two-dimensional feature image; then, the CNN-reBert and 2D-ResNet18 network modules are used to extract the feature information in the one-dimensional range image and the two-dimensional feature image respectively, and splice and fuse them into new feature information; finally, the attention layer, two fully connected layers and softmax are used to output the final result to complete the target classification. By comparing with a variety of existing models, the experimental results show that Rf-BERT can achieve the best recognition accuracy under different signal-to-noise ratio noises, proving the effectiveness of the model.

[0168] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A sea surface target recognition method based on multimodal fusion, characterized by: The target recognition method includes the following steps: Step 1: GAF angular field: GAF can convert time series data into image data, which can not only retain the complete information of the signal but also maintain the dependence of the signal on time; after the signal data is converted into image data, the advantages of 2D-CNN in image classification and recognition can be fully utilized to extract features; Step 2: Rf-BERT network: Use the Gram angle and field in GAF to convert the one-dimensional range profile into a two-dimensional feature image; then use the CNN-reBert and 2D-ResNet18 network modules to extract the feature information in the one-dimensional range profile and the two-dimensional feature map respectively, and splice and fuse them into new feature information; finally, use the attention layer, two fully connected layers and softmax to output the probability distribution to complete target classification; Step 3: Data and experimental analysis.

2. The method for sea surface target recognition based on multimodal fusion according to claim 1, characterized in that: The idea of the said Step 1 mainly comes from the concept of the Gram matrix in linear algebra; a matrix composed of the inner products of any k (k < n) vectors in an n-dimensional Euclidean space is called a Gramian matrix; the Gramian matrix can reflect the linear correlation between vectors; it can be expressed as: where u1,u2,...,u k are all n-dimensional vectors; the inner product of each vector can be expressed as: HRRP is a combination of distance units and can be expressed as X = [x1, x2, ..., x n ], where X represents the entire HRRP data and n represents the data length; for HRRP data, the first step is to scale X to [-1, 1] or [0, 1], which can be achieved by formula (3) or (4), respectively: or After scaling, it can prevent the inner product from tending to be too large originally; the Gramian matrix of HRRP can be expressed as: Therefore, the HRRP time series can be recalibrated using polar coordinates Encode the value as the cosine of the angle and the timestamp as the radius; the formula is as follows: Among them, t i is the timestamp of the moment, r i It is a polar axis, which preserves the temporal relationship; is the polar angle, retaining the numerical relationship; First, scale HRRP to [-1, 1] through formula (3), then convert the Cartesian coordinate system to the polar coordinate system through formula (6); finally, the time correlation within different time intervals can be identified from the perspective of angles by using the sum or difference of triangles between each point, and a Gramian-like matrix that maintains time dependence is generated; there are two methods, the Gram sum angular field and the Gram difference angular field, to generate the matrix, which are defined as follows: Through the GAF algorithm, the one-dimensional HRRP is converted into a two-dimensional image, thus applying the technology in the field of computer vision to the research of HRRP.

3. The method for sea surface target recognition based on multimodal fusion according to claim 1, characterized in that: The said Step 2 includes the following: A1: ResNet-18: ResNet alleviates the degradation problem in deep networks through the residual structure; CNN-reBert module: In ResNet, a residual block can be expressed as: y=F(x,{W i })+x (9) Where x and y are the input and output vectors of the layers under consideration, and W i is the weight matrix, and F is the residual function; F can be expressed as follows: F = W2g(W1x + b1) + b2 (10) where W1, W2 are weight matrices, b1 and b2 are biases, and g represents the activation function ReLU; A2: CNN-reBert module: In the CNN-reBert module, the number of channels of the input one-dimensional HRRP is first increased to 512, and a 1×5 convolution kernel is used for one-dimensional convolution; on the one hand, the extracted sequence features better retain the local structural information in HRRP; on the other hand, the dimension and number of channels of the convolution kernel independently control the dimension and sequence length of the features, respectively, realizing the decoupling of the two and improving the whole network’s ability to represent HRRP data; in addition, the size and stride of the convolution kernel are controllable, avoiding fixed position constraints and enhancing the initial expression ability of the network; position encoding is used to add timing information to the one-dimensional distance image, avoiding the problem that the cyclic structure is difficult to parallelize; in CNN-reBert, a total of 4 layers of reBERT models are stacked, namely N bert =4, using the 8 multi-head attention and forward connection modules in the Bert module to effectively extract the radar HRRP features and provide better input for the classification network; A3: Hyperparameters.

4. The method for sea surface target recognition based on multimodal fusion according to claim 3 is characterized in that: The said A2 includes the following: A21: Self-attention mechanism: The self-attention mechanism can perform parallel calculations using matrix multiplication, with the characteristics of faster speed and higher spatial efficiency; The sequence feature F after position encoding emb Perform three transformations respectively to obtain d q The query matrix Q of dimension d k The key matrix K and d v Dimensional value matrix V: Q=F emb W q K=F emb W k V=F emb W v in d q =d k ;d model is the length of the input sequence features; The multiplication of Q and K generates an attention map, representing the correlation between all tokens within each layer; Divide by the scaling factor Finally, the attention weight of V is obtained through the softmax function, and its formula is: A22: Multi-head self-attention mechanism: HRRP data has a typical multi-subspace distribution structure, and target structure information can be obtained from multiple subspaces; use the multi-head attention mechanism to map the input to multiple subspaces and deeply extract information helpful for recognition from multiple dimensions; Perform h linear projections on the matrices Q, K, and V, that is, project them into h different subspaces and calculate them separately; then concatenate all the calculation results, and finally perform linear transformation to obtain the output F multi : F multi =Concat(head1,head2,...,head h )W O (12) Where h is the number of longs, are three sets of weight matrices, is the output projection matrix; A23: Re-attention: Use the head's attention map as the basis and generate a set of new attention maps through dynamic aggregation; for this purpose, a learnable transformation matrix θ∈R is defined h×h , and then mix the multi-head attention map into the regenerated new map before multiplying it with V; the original self-attention and re-attention; Re-attention replaces the overly simple trade-off of dropout, and the implementation is simple, the formula is as follows: Among them, the transformation matrix θ multiplies the self-attention map along the head dimension; here, Norm is a normalization function used to reduce the variance layer by layer; θ can be learned end-to-end; A24: Attention layer: The attention layer can dynamically weight the fused features. In order to ensure that the fused features useful for recognition can be used more efficiently, we introduce the attention mechanism to the feature sequence O of the fused output. bert Weighted along the feature dimension, we get feature F att : Among them, O bert (k,l) is the lth element in the kth channel feature of the fused output feature, and α(k,l) represents the weight of the corresponding element.

5. The method for sea surface target recognition based on multimodal fusion according to claim 3, characterized in that: The said A3 includes the following: A31: Loss function: The loss function is the cross entropy loss function (CrossEntropy Loss) which is currently widely used in classification tasks; in Pytorch, you can directly call CrossEntropyLoss, and its calculation formula is as follows: The log has base e; the first parameter x is the input and the output of the last layer of the network, and its shape is [batchsize, class]; To get the final cross entropy loss in Pytorch, you also need to go through the loss weight, which is calculated as follows: loss(x,class)=weight[class](-x[class]+log(∑exp(x[i]))) (16) A32: Optimizer: Use Adam optimizer; A33: Learning rate: Use the OneCycleLR method to adjust the learning rate, that is, increase the learning rate in the initial training process to make the model learn faster, and slowly reduce the learning rate at a certain stage of training to allow the model to find the global optimal solution.

6. The method for sea surface target recognition based on multimodal fusion according to claim 1, characterized in that: The step three includes the following: B1: Data preprocessing: Preprocessed data will make it easier to find the minimum value during the iteration of the gradient descent method, which will speed up the model convergence. Preprocessing includes the following two steps: B11: Normalize the amplitude of HRRP; the data after normalization of the nth HRRP amplitude is expressed as where max(|x n |) represents the maximum absolute value of all elements in HRRP; B12: subtract the mean of the normalized HRRP data from the respective elements; B2: Dataset: There are two methods to obtain target echo signals, namely, actual measurement and theoretical calculation. Since most ship targets are non-cooperative targets, it is very difficult to obtain HRRP by field measurement. Six ship models were established using 3D Max, and HRRP was calculated using FEKO. The simulation parameters set included the radar center frequency of 10 GHz, bandwidth of 80 MHz, number of frequency sampling points of 200, azimuth range of 0-360°, interval of 1°, and grazing angle of 10°. The HRRP obtained has 200 range cells, and the length of each range cell is 1.875 m. In order to meet the sample data volume requirements during neural network training and prevent overfitting, the data set needs to be expanded; two expansion methods are selected: B21: Data enhancement for HRRP; when calculating HRRP, the coordinate axis coincides with the center of the ship, so the effective HRRP information is generally in the middle area; in actual environments, when the radar detects a target, the echo signal may be incomplete or partially lost; Using data augmentation technology, the dataset is expanded by shifting the HRRP signal along the distance direction, 20 distance units each time, twice to the left and twice to the right; deleted data is discarded, and the blank part is filled with 0 using zero-padding; by taking those overlapping but different HRRP samples, the number of samples is increased by 4 times; B22: Add random noise to the translated HRRP data; add 5 times Gaussian white noise to the data, and the data after adding noise meets a certain signal-to-noise ratio; The expanded data set is divided into a training set and a test set according to the ratio of 8:2 for each type of ship target. The samples of the training set and the test set are 43200 and 10800 respectively. Since this paper adopts the supervised learning method, in the process of data set division, the category label of the data should be constructed at the same time, and each label should correspond to each group of data one by one. In addition, in order to ensure the authenticity and effectiveness of the classification and recognition performance of the network model, the entire data set needs to be randomly shuffled and reorganized before division. B3: Comparative experiment: SVM has the highest recognition accuracy of 94.89% for TUO, but the lowest recognition accuracy of 80.83% for LCS, and the overall weighted accuracy is 89.96%; CNN has the highest recognition accuracy of 90.11% for CVN-65, but the lowest recognition accuracy of 74.5% for YOU, and the overall weighted recognition rate is 82.43%; the weighted accuracy of TATDRNN and LSTM are close, 93.79% and 94.87% respectively, but both have a probability of more than 3% to mistake TUO for YOU, LCS and DDG104; CNN-BERT has a high overall weighted accuracy of 96.9%, and the recognition accuracy of each category is also high; and the overall weighted accuracy of Rf-BERT reaches 98%, and compared with CNN-BERT, the recognition of each category is more accurate; B4: Ablation experiment: When only the original Transformer encoder is used for classification, the weighted accuracy is only 94.32%; but after using a simple convolution block to extract features in advance, the accuracy will increase a little to 95.28%; then after adding position encoding to the features extracted by the convolution block, the weighted accuracy will reach 95.42%; if only position encoding is added to the original Transformer encoder, the weighted accuracy can reach 94.68%; replace the ordinary multi-head self-attention with Re-attent ion, the accuracy will increase slightly to 95.53%, but it can effectively avoid attention collapse; using the pre-trained two-dimensional Resnet18 alone to classify and recognize the GAF images converted from HRRP, the accuracy can reach 97.02%; if the HRRP features extracted by the previous CNN-reBert module are added for recognition and classification, the overall weighted accuracy will be slightly improved to 97.63%; finally, with the addition of the attention layer, the complete Rf-BERT will be assembled, and the overall weighted recognition accuracy can reach 98.06%.

Images