Multi-view SAR target identification method based on multi-scale feature fusion of feature pyramid

Abstract

The invention discloses a multi-view SAR (synthetic aperture radar) target recognition method based on multi-scale feature fusion of a feature pyramid, which comprises the following steps of: (1) improving a Resnet50 feature extraction network, and extracting multi-scale feature information from four residual structures of the network; (2) fusing multi-scale features by using an improved feature pyramid network FPN, and proposing a residual feature pyramid network ResFPN; (3) combining the method with a multi-view method, and proposing a multi-view residual error characteristic pyramid network MV-ResFPN; and (4) training the MV-ResFPN network to obtain a training weight, and realizing multi-view SAR target identification. According to the method, the Resnet feature extraction network is improved from the aspect of multi-scale feature fusion and is combined with multiple views, the multi-view residual feature pyramid network MV-ResFPN is provided, fusion of shallow features and deep features is enhanced, the problem of SAR image angle sensitivity is solved, and the recognition precision of network complex scene targets is improved.

Description

Technical Field

[0001] This invention relates to the fields of SAR image processing and target recognition technology, specifically to a multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion. Background Technology

[0002] Synthetic Aperture Radar (SAR) is a radar system operating in the high-resolution microwave band. It actively transmits and receives microwaves and uses signal processing methods for imaging. As a microwave imaging radar, SAR possesses unique advantages such as all-weather capability, high resolution, and long-range operation, making it extremely important in defense, military, geological, and natural disaster monitoring. In recent years, researchers have obtained a large number of high-resolution SAR images, making the processing of these images a pressing problem. Automatic target recognition in SAR has always been a research hotspot and one of its most severe challenges. Traditional SAR automatic recognition methods mainly utilize simple feature extraction techniques to distinguish target categories, such as geometric and scattering features. Although these methods have achieved good performance in certain scenarios, they often require manually designed features and complex mathematical theories. These features are time-consuming, labor-intensive, and have poor generalization ability, making them insufficient for processing large amounts of information.

[0003] Deep learning networks have demonstrated exceptional performance in text translation, computer vision, and natural language processing tasks, significantly enhancing information extraction capabilities. Therefore, researchers have begun applying deep learning to extract feature information from SAR images to improve data representation. However, most existing methods rely on image intensity information, extracting only deep features, which often leads to overfitting due to limited training samples. Furthermore, the angular sensitivity of SAR images, coupled with complex backgrounds accompanied by high noise and occlusion, results in significant intra-class differences but insignificant inter-class differences. Therefore, improving target recognition rates in such complex scenarios is a crucial problem that must be addressed. Summary of the Invention

[0004] Purpose of the invention: This invention provides a multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion, which solves the problem of fusing shallow contour features and deep semantic features, and solves the problems of large intra-class differences and insignificant inter-class differences in SAR images caused by complex scenes and angle sensitivity.

[0005] Technical solution: The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion described in this invention specifically includes the following steps:

[0006] (1) Extract four multi-scale feature information from the four residual structures of the ResNet50 feature extraction network;

[0007] (2) The improved feature pyramid network FPN is used to fuse multi-scale features. The first three multi-scale features are fused from high to low levels to obtain two fused features. The original output features of ResNet and the fused multi-scale features are adaptively weighted. While retaining the original output of ResNet, the correlation between shallow and deep features is learned. The residual feature pyramid network ResFPN is constructed.

[0008] (3) Combine the Residual Feature Pyramid Network (ResFPN) with the multi-view method to obtain the Multi-view Residual Feature Pyramid Network (MV-ResFPN) and fuse target feature information from different perspectives;

[0009] (4) Train the MV-ResFPN network to obtain training weights and realize the identification of multi-view SAR targets.

[0010] Furthermore, the implementation process of step (1) is as follows:

[0011] The ResNet50 feature extraction network has four residual modules. A Bottleneck structure is introduced into each residual module to extract the output of each residual module separately, resulting in four multi-scale features.

[0012] Furthermore, the implementation process of step (2) is as follows:

[0013] After convolution, all channels of the multi-scale features output by the four residual networks are converted to 256 to align the channel dimensions.

[0014] For the first three multi-scale features f1, f2 and f3, they were fused from the high level to the low level. f3 was upsampled to 32×32 by the nearest neighbor interpolation method and added to f2 to obtain f2', with a feature size of 32×32.

[0015] f2' is upsampled to 64×64 using the nearest neighbor interpolation method, and then added to f1 to obtain f1', with a feature size of 64×64;

[0016] f4 is expanded to a feature size of 32×32 by transposing convolution to obtain f4'; f1' is downsampled to 32×32; f1', f2' and f4' are adaptively weighted by an adaptive weighted averager to fuse multi-scale target information;

[0017] The final fused features are obtained by dimensionality reduction through a global average pooling layer.

[0018] Furthermore, the implementation process of step (3) is as follows:

[0019] For each image, two other images under the same label are randomly extracted to form a set of images. After passing through three parallel ResFPNs, three multi-view features are obtained. An LSTM network for processing time series is introduced to concatenate the features in the channel dimension and map the angle dimension to the time dimension. The concatenated features are then fed into the LSTM network as a sequence for multi-view fusion. The fused features are then passed through a Softmax classifier for SAR target prediction and recognition.

[0020] Furthermore, the four residual modules respectively include 3, 4, 6 and 3 residual structures.

[0021] Furthermore, the convolution size is 1x1.

[0022] Furthermore, the transposed convolution size is 4×4.

[0023] Furthermore, the hidden layer size of the LSTM network is set to 256.

[0024] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] This invention proposes the ResFPN method to fuse multi-scale feature information of a target. By fusing multi-scale features obtained from a residual network, it integrates shallow contour features with deep semantic features without ignoring the features output by ResNet itself, thereby enhancing feature representation capabilities and increasing inter-class diversity. Then, combined with a multi-view approach, a multi-view residual feature pyramid network MV-ResFPN is proposed, which strengthens the completeness of target information from different viewpoints, solves the problem of large intra-class differences caused by angle sensitivity, and thus improves the network's recognition rate of targets in complex backgrounds. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention;

[0027] Figure 2 A schematic diagram of the multi-view residual feature pyramid network MV-ResFPN;

[0028] Figure 3 This is a schematic diagram of the Resnet50 network structure;

[0029] Figure 4 A schematic diagram of the improved three-layer feature pyramid network (FPN) structure;

[0030] Figure 5 This is a schematic diagram of the Long Short-Term Memory (LSTM) network structure. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings.

[0032] like Figure 1 As shown, this invention proposes a multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion, comprising the following steps:

[0033] Step 1: Improve the feature extraction network of ResNet50 to extract multi-scale feature information from the four residual structures of the network.

[0034] like Figure 2 As shown, the ResNet network has four modules. Extracting the output of each module separately yields four multi-scale features. For example... Figure 3 The diagram shows a ResNet50 architecture with an input image size of 112×112. The multi-scale features extracted by the ResNet50 feature extraction network consist of four residual modules, each containing 3, 4, 6, and 3 residual structures respectively. The residual structures incorporate a Bottleneck structure. The features output by each residual module are extracted. For an input image of 128×128, the features output by the four layers are denoted as f1, f2, f3, and f4, with feature sizes of 64×64, 32×32, 16×16, and 8×8 respectively.

[0035] Step 2: Improve the Feature Pyramid Network (FPN) by fusing multi-scale features and adding an adaptive weighting network to propose the Residual Feature Pyramid Network (ResFPN).

[0036] like Figure 2 As shown, this invention improves the feature pyramid network to fuse multi-scale features. The specific implementation process of the residual feature pyramid network is as follows: after a 1x1 convolution, all channels of the multi-scale features output by the four residual networks are converted to 256 to align the channel dimensions. For the first three multi-scale features f1, f2, and f3, as shown... Figure 4As shown, fusion is performed from high to low layers. f3 is upsampled to 32×32 using nearest neighbor interpolation and added to f2 to obtain f2', with a feature size of 32×32. f2' is upsampled to 64×64 using nearest neighbor interpolation and added to f1 to obtain f1', with a feature size of 64×64. f4 is expanded to 32×32 using a 4×4 transposed convolution to obtain f4'; f1' is downsampled to 32×32; f1', f2', and f4' are adaptively weighted using an adaptive weighted averager to fuse multi-scale target information. Dimensionality reduction is achieved through a global average pooling layer to obtain the final fused features. This invention, by comparing bilinear interpolation and nearest neighbor interpolation, finds that bilinear interpolation makes the image smoother, while nearest neighbor interpolation better highlights features and performs better in feature processing; therefore, choosing nearest neighbor interpolation is more reasonable. This improved residual feature pyramid network retains the original output f4 of ResNet through adaptive weighting, while fusing feature information from low to high layers, thus integrating shallow contour information with deep semantic information.

[0037] Step 3: By combining the Residual Feature Pyramid Network (ResFPN) with the multi-view method, a multi-view residual feature pyramid network (MV-ResFPN) is proposed, which integrates target feature information from different perspectives.

[0038] This invention integrates a multi-scale feature fusion network with multiple perspectives. For example... Figure 5 As shown, the multi-view fusion method used in this invention is a Long Short-Term Memory (LSTM) network. The multi-view residual feature pyramid network specifically works as follows: for each image, two other images under the same label are randomly extracted to form a group of images. These groups are then processed by three parallel ResFPN networks to obtain three multi-view features. An LSTM network for processing time series is introduced to concatenate the features along the channel dimension, mapping the angle dimension to the time dimension. The concatenated features are then fed into the LSTM network as a sequence for multi-view fusion. The hidden layer size of the LSTM network is set to 256. The fused features are then processed by a Softmax classifier for prediction and recognition. By mapping from angle to time, target feature information from different angles can be effectively fused, solving the problem of large intra-class differences in SAR images caused by angle sensitivity. The MV-ResFPN network is trained to obtain training weights, enabling the recognition of multi-view SAR targets.

[0039] To verify the effectiveness of this invention, two experiments were conducted. In the first experiment, ResNet, ResFPN, and MV-ResFPN networks were first trained on samples with a 17° pitch angle from the MSTAR dataset. Then, after training, the networks were tested on samples with a 15° pitch angle from the MSTAR dataset, each with its own training weights. The distribution of training and testing samples in the MSTAR dataset is shown in Table 1. The experiment lasted 100 epochs, with an input image size of 128×128. The Adam algorithm with a batch size of 16 was used for optimization, amsgrad was set to True, weight decay was set to 0.001, and the learning rate was set to a dynamic learning rate, initially 0.0001, adjusted to 0.9 times the previous epoch after each epoch. The experimental results are shown in Table 2, specifically: the recognition rate obtained by ResNet was 95.26%, the recognition rate obtained by ResFPN was 97.69%, and the recognition rate obtained by MV-ResFPN was 99.96%. This demonstrates that the residual feature pyramid module and the multi-view fusion module significantly improve the training effect of the network, thereby increasing the accuracy of target recognition.

[0040] Table 1 shows the distribution of training and testing samples in the MSTAR dataset.

[0041] category Number of training samples Number of test samples BMP2 233 196 BTR70 233 196 T72 232 196 T62 299 273 BRDM2 298 274 BTR60 256 195 ZSU23 299 274 D7 299 274 ZIL131 299 274 2S1 299 274

[0042] Table 2 compares the recognition rates of ResNet, ResFPN, and MV-ResFPN on the MSTAR dataset.

[0043] method Recognition rate (%) Resnet 95.26 ResFPN 97.69 MV-ResFPN 99.96

[0044] The second comparative experiment involved the following steps: First, a self-captured circular SAR dataset with more complex backgrounds was selected. ResNet, ResFPN, and MV-ResFPN networks were used to train the training samples. Then, after training, each network was tested on the same samples with its respective training weights. The distribution of training and testing samples for the circular SAR dataset is shown in Table 3. The experiment lasted 100 epochs, with an input image size of 128×128. The Adam algorithm with a batch size of 16 was used for optimization, amsgrad was set to True, weight decay was set to 0.001, and the learning rate was set to a dynamic learning rate, initially 0.0001, adjusted to 0.9 times the previous epoch after each epoch. The experimental results are shown in Table 4. Specifically, ResNet achieved a recognition rate of 72.45%, ResFPN achieved 84.84%, and MV-ResFPN achieved 92.31%, representing an improvement of nearly 20% compared to ResNet. This demonstrates that in recognition tasks with more complex datasets, the multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion significantly improves the target recognition accuracy and performs better.

[0045] Table 3 shows the distribution of training and testing samples in the circular SAR dataset.

[0046] category Number of training samples Number of test samples 54TH 1094 446 59-1TC 1105 854 59AG 1093 2027 62LT 1103 256 63APTV 1101 414 63AT 1102 488 63CAAV 1104 357 J6 1101 416 T34-85 1084 1810

[0047] Table 4 compares the recognition rates of ResNet, ResFPN, and MV-ResFPN on the circular SAR dataset.

[0048] method Recognition rate (%) Resnet 72.45 ResFPN 84.84 MV-ResFPN 92.31

[0049] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion, It is characterized in that The steps include: (1) Extract four multi-scale feature information from the four residual structures of the feature extraction network of Resnet50; (2) The improved feature pyramid network FPN is used to fuse multi-scale features. The first three multi-scale features are fused from high to low layers to obtain two fused features. The original output features of Resnet and the fused multi-scale features are adaptively weighted. While retaining the original output of Resnet, the association between shallow and deep features is learned; the residual feature pyramid network ResFPN is constructed; (3) Combine the residual feature pyramid network ResFPN with the multi-view method to obtain the multi-view residual feature pyramid network MV-ResFPN, which integrates the target feature information from different angles; (4) Train the MV-ResFPN network to obtain training weights and realize multi-view SAR target recognition.

2. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 1, It is characterized in that The implementation process of step (1) is as follows: The Resnet50 feature extraction network has a total of four residual modules. A Bottleneck structure is introduced in each residual structure, and the output of each residual module is extracted separately to obtain four multi-scale features.

3. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 1, It is characterized in that The implementation process of step (2) is as follows: After the convolution operation, all channels of the multi-scale features output by the four residual networks are converted to 256 to align the channel dimensions; For the first three multi-scale features f1, f2 and f3, they are fused from high level to low level. f3 is upsampled to 32×32 by the nearest neighbor interpolation method and added to f2 to get f2', with a feature size of 32×32. Upsample f2' to 64×64 using the nearest neighbor interpolation method and add it to f1 to get f1', with a feature size of 64×64; f4 is transposed convolutionally expanded to 32×32 to obtain f4'; f1' is downsampled to 32×32; f1', f2' and f4' are adaptively weighted by an adaptive weighted averager to fuse multi-scale target information; The final fusion feature is obtained by reducing the dimension through a global average pooling layer.

4. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 1, It is characterized in that The implementation process of step (3) is as follows: For each image, two other images with the same label are randomly extracted to form a group of images. After three parallel ResFPNs, three multi-view features are obtained. An LSTM network for processing time series is introduced to splice the features in the channel dimension and map the angle dimension to the time dimension. After splicing, the features are sent to the LSTM network as a sequence for multi-view fusion. The fused features are passed through the Softmax classifier for SAR target prediction and recognition.

5. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 2, It is characterized in that The four residual modules include 3, 4, 6 and 3 residual structures respectively.

6. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 3, It is characterized in that The convolution size is 1x1.

7. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 3, It is characterized in that The transposed convolution size is 4×4.

8. The multi-view SAR target recognition method based on feature pyramid multi-scale feature fusion according to claim 4, It is characterized in that The hidden layer size of the LSTM network is set to 256.

Images