Deep neural network target recognition system and method based on multi-source sar image fusion

Abstract

The application relates to a kind of deep neural network target identification methods based on multi-source SAR image fusion: multi-source SAR image is fused, the image information that single image SAR image cannot shoot is made up, and super-definition fusion image is obtained;Deep neural network identification model is used, and the target identification of SAR image is carried out to super-definition fusion image.Multi-source SAR image fusion method makes up the image information that single SAR image cannot shoot due to the obstruction of cloud, building and other factors in shooting direction, realizes providing feature-rich, clear input image for deep neural network target identification method, compared with the image recognition technology based on single direction shooting image, has higher image recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of Synthetic Aperture Radar (SAR) image recognition, and more particularly to a deep neural network-based target recognition method based on multi-source SAR image fusion. Background Technology

[0002] SAR is an active microwave remote sensing imaging device that can be installed on flight platforms such as aircraft, satellites, and spacecraft to conduct all-day, all-weather observations of the Earth. It has a wide range of applications in monitoring, navigation, and remote sensing, and its research value is extremely significant.

[0003] SAR images differ from optical images in their imaging principles. SAR is a coherent imaging system. Therefore, speckle noise inevitably appears in SAR images. This speckle noise severely disturbs the edge and texture information of targets in SAR images, thus seriously affecting the recognition of SAR images.

[0004] In recent years, deep learning and artificial neural networks in the field of artificial intelligence have made significant progress in the recognition, detection, and segmentation of optical images, becoming a current research hotspot. Some researchers have gradually introduced them into the field of SAR image target recognition. However, because SAR images are grayscale images, lacking the rich image features of optical images, and suffering from severe speckle noise interference, even the most advanced deep artificial neural network recognition algorithms face numerous challenges. Deep neural network algorithms for target recognition not only rely on extracting as many features as possible from SAR images but also depend on training with a massive dataset of SAR images. However, factors such as speckle noise, azimuth, and elevation angles in SAR images severely affect feature extraction; furthermore, the high cost of acquiring SAR data results in an insufficient number of SAR image samples, thus hindering the performance of current deep neural network algorithms in SAR image recognition. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a deep neural network target recognition method based on multi-source SAR image fusion. Based on limited SAR image data, it reduces the influence of factors such as speckle noise, azimuth angle, and elevation angle, and performs SAR image target recognition algorithm more accurately and effectively.

[0006] The solution of this invention is: a deep neural network target recognition method based on multi-source SAR image fusion, which includes the following steps:

[0007] S1. Fusion of multi-source SAR images to compensate for image information that cannot be captured by a single SAR image, resulting in an ultra-high-definition fused image;

[0008] S2. A deep neural network recognition model is used to perform target recognition of SAR images on ultra-high-definition fused images.

[0009] Preferably, the multi-source SAR images include multi-directional, multi-azimuth, and multi-scale SAR images. Multi-directional SAR images refer to SAR images taken by different cameras from different angles. Multi-azimuth SAR images refer to SAR images taken by the same camera at different orientations. Multi-scale SAR images refer to SAR images of different pixel sizes taken by the same camera at different distances while maintaining the same orientation.

[0010] Preferably, the specific method of step S1 is as follows:

[0011] S1-1. Acquire multi-source SAR images. Correct the coordinates of SAR images taken from multiple directions and at multiple scales in the same area and unify them into the same coordinate system.

[0012] S1-2, Unify and adjust SAR images from multiple directions, orientations, and scales into SAR images to be fused with the same pixel size;

[0013] S1-3. Extract image features from the SAR images to be fused using a convolutional neural network (CNN) to obtain a feature map for each SAR image to be fused; the feature map includes the shape features and texture features of the target.

[0014] S1-4. Use a fusion rule network to perform image feature-level fusion on each feature map to obtain an ultra-high-definition fused image.

[0015] Preferably, the image feature-level fusion includes three steps: neighboring image sorting, key feature point matching, and ultra-high-definition fused image generation.

[0016] Adjacent feature map sorting: Sort multiple feature maps according to the principle of minimizing the sum of the eigenvalue distances of adjacent feature maps to obtain a sequence of adjacent feature maps;

[0017] Key feature point matching: Multiple feature maps are matched and superimposed to obtain a fused dense feature map;

[0018] Ultra-high definition fused image generation: The fused dense feature map is subjected to cross-validation of adjacent pixels, elimination of redundant information, removal of outliers, stitching of blind spots, mapping using a global color model, and vignetting correction to obtain an ultra-high definition fused image.

[0019] Preferably, the method for sorting adjacent graphs is as follows:

[0020] S1a. Randomly select a feature map as the reference feature map and include the reference feature map in the adjacent feature map sequence.

[0021] S2a. Denote the feature maps outside the adjacent feature map sequences as candidate feature maps;

[0022] S3a. Calculate the similarity between each candidate feature map and the baseline feature map, and determine the candidate feature map with the smallest similarity to the baseline feature map as the neighboring map of the baseline feature map.

[0023] S4a. Sort the adjacent graphs according to their similarity from smallest to largest as the sorting basis, and include the adjacent graphs of the benchmark feature graph in the adjacent feature graph sequence;

[0024] S5a. Select a neighboring map of the reference feature map as the new reference feature map, and repeat steps S2a to S5a until all feature maps have been traversed.

[0025] Preferably, the key feature point matching step is as follows:

[0026] S1b: Extract the feature map with the largest sum of the absolute values ​​of all pixel feature values ​​in adjacent feature map sequences, and use it as a reference feature map;

[0027] S2b: Extract the key pixels from the reference feature map, set the feature values ​​of the remaining pixels to zero, and obtain the master template to be matched.

[0028] S3b. Extract another feature map from the adjacent feature map sequence and denote it as the sub-board to be matched. Perform sliding window processing on the sub-board to be matched along the four directions of up, down, left, and right with a preset step size. Calculate the similarity between the sub-board to be matched after multiple sliding windows and the master board to be matched. Denote the sub-board to be matched after the sliding window with the smallest similarity to the master board to be matched as the feature map to be matched.

[0029] S4b: Repeat step S3b until the remaining feature maps in the adjacent feature map sequence are traversed, then proceed to step S5b.

[0030] S6b: Superimpose all feature maps to be matched with the corresponding feature values ​​of the master image to be matched to generate a fused dense feature map.

[0031] Preferably, the similarity calculation method for two feature maps is as follows:

[0032] The distance between the eigenvalues ​​of the two feature maps is obtained by subtracting their corresponding eigenvalues ​​and taking their absolute values. The sum of the distances between the eigenvalues ​​of the two feature maps is the similarity between them.

[0033] Preferably, the deep neural network recognition model includes an image feature extraction network, a small-scale target extraction network, a region proposal network, a multi-scale region of interest pooling layer, a connection layer, and a target recognition output layer;

[0034] An image feature extraction network is used to extract edge, contour, and texture features from the generated fused image to obtain a target recognition feature map.

[0035] The small-scale target extraction network further extracts small-scale target image features from the target recognition feature map, resulting in an image feature map that includes small-scale target edges, contours, and textures.

[0036] The region proposal network is used to generate 2D bounding boxes for multiple regions of interest on the image feature map, and to use these generated 2D bounding boxes to segment the sub-image feature map within the corresponding 2D bounding box on the image feature map.

[0037] The feature maps of different sizes of sub-images segmented from the 2D bounding box are pooled by region of interest to generate a unified image feature map of a fixed size.

[0038] The fixed-size image feature map generated uniformly after pooling is shared to the fully connected layer for SAR image target classification and 2D bounding box regression, thus completing the target recognition of the SAR image. The recognition result is output through the target recognition output layer.

[0039] Preferably, the image feature extraction network is a ResNet-50 network.

[0040] Preferably, the small-scale target extraction network is a feature pyramid network.

[0041] The beneficial effects of this invention compared to the prior art are:

[0042] (1) By adopting a multi-source SAR image fusion method, this invention provides a feature-rich and clear input image for deep neural network target recognition methods. Compared with existing image recognition techniques based on images taken from a single direction, this invention has a higher image recognition accuracy.

[0043] (2) The present invention proposes a multi-source SAR image fusion method to make up for the image information that cannot be captured by a single SAR image due to the obstruction of factors such as clouds, fog and buildings in the shooting direction;

[0044] (3) This invention solves the problem that SAR images with rich and clear features cannot be obtained due to factors such as speckle noise, azimuth angle, and elevation angle by using multi-source SAR images based on image feature level image fusion.

[0045] (4) The present invention proposes a deep neural network target recognition method based on multi-source SAR image fusion to solve the problem of poor SAR image target recognition effect based on limited SAR image data. Attached Figure Description

[0046] Figure 1This is a schematic diagram of a multi-source SAR image fusion method based on a convolutional neural network (CNN) according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of a deep neural network target recognition method based on multi-source SAR image fusion according to an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0049] By leveraging techniques such as multi-scale geometric analysis, image spatial filtering, sparse representation, and deep learning, modern image fusion can utilize advanced image processing technologies to fuse multi-source images, resulting in more information-rich and clearer images that better meet practical needs. Therefore, given that multi-source SAR image fusion can achieve certain noise suppression and provide richer image features, this invention proposes a deep neural network target recognition method based on multi-source SAR image fusion. This method includes the following steps:

[0050] S1. Fusion of multi-source SAR images to compensate for image information that cannot be captured by a single SAR image, resulting in an ultra-high-definition fused image;

[0051] S2. A deep neural network recognition model is used to perform target recognition of SAR images on ultra-high-definition fused images.

[0052] As can be seen, the method of the present invention mainly includes two parts: one is the multi-source SAR image fusion algorithm, and the other is the deep neural network recognition algorithm.

[0053] 1. Multi-source SAR image fusion algorithm

[0054] Figure 1 This is a schematic diagram of the multi-source SAR image fusion method based on convolutional neural networks (CNN) proposed in this invention. As shown in Figure 1, this invention introduces CNN into multi-source SAR image fusion. The CNN-based image fusion method has the ability to learn features hierarchically, resulting in more diverse feature representations, stronger discriminative performance, and better generalization performance.

[0055] The multi-source SAR images include multi-directional, multi-azimuth, and multi-scale SAR images. Multi-directional SAR images refer to SAR images taken by different cameras from different angles. Multi-azimuth SAR images refer to SAR images taken by the same camera at different orientations. Multi-scale SAR images refer to SAR images of different pixel sizes taken by the same camera at different distances while maintaining the same orientation.

[0056] To obtain a complete fused image in a unified coordinate system, all feature maps need to be fused at the image feature level based on key edges, contours, and texture features through a fusion rule network to obtain the final ultra-high-definition fused image.

[0057] The specific steps of the CNN-based image fusion algorithm are as follows:

[0058] S1-1. Acquire multi-source SAR images (SAR image 1, SAR image 2, ...). Correct the coordinates of SAR images taken from multiple directions and at multiple scales in the same area and unify them into the same coordinate system.

[0059] S1-2. Adjust SAR images from multiple directions, orientations, and scales into a single SAR image to be fused with the same pixel size; the pixel size is preferably 1000*1000.

[0060] S1-3. Extract image features from the SAR images to be fused using a convolutional neural network (CNN) to obtain a feature map for each SAR image to be fused. The feature map includes the shape features and texture features of the target. The shape features include edges, contours, etc. The texture features include the thickness and density of the texture, etc.

[0061] The image feature-level fusion includes three steps: neighbor image sorting, key feature point matching, and ultra-high-definition fused image generation.

[0062] Adjacent feature map sorting: Sort multiple feature maps according to the principle of minimizing the sum of the eigenvalue distances of adjacent feature maps to obtain a sequence of adjacent feature maps;

[0063] The method for sorting adjacent graphs is as follows:

[0064] S1a. Randomly select a feature map as the reference feature map and include the reference feature map in the adjacent feature map sequence.

[0065] S2a. Denote the feature maps outside the adjacent feature map sequences as candidate feature maps;

[0066] S3a. Calculate the similarity between each candidate feature map and the baseline feature map, and determine the candidate feature map with the smallest similarity to the baseline feature map as the neighboring map of the baseline feature map.

[0067] The similarity between two feature maps is calculated as follows:

[0068] The distance between the eigenvalues ​​of the two feature maps is obtained by subtracting their corresponding eigenvalues ​​and taking their absolute values. The sum of the distances between the eigenvalues ​​of the two feature maps is the similarity between them.

[0069] S4a. Sort the adjacent graphs according to their similarity from smallest to largest as the sorting basis, and include the adjacent graphs of the benchmark feature graph in the adjacent feature graph sequence;

[0070] S5a. Select a neighboring map of the reference feature map as the new reference feature map, and repeat steps S2a to S5a until all feature maps have been traversed.

[0071] Key feature point matching: Multiple feature maps are matched and superimposed to obtain a fused dense feature map;

[0072] The key feature point matching steps are as follows:

[0073] S1b: Extract the feature map with the largest sum of the absolute values ​​of all pixel feature values ​​in adjacent feature map sequences, and use it as a reference feature map;

[0074] S2b: Extract key feature points from the reference feature map, set the feature values ​​of the remaining feature points to zero, and obtain the master template to be matched; key feature points are selected from those where the feature values ​​change drastically.

[0075] S3b. Extract another feature map from the adjacent feature map sequence and denote it as the sub-board to be matched. Perform sliding window processing on the sub-board to be matched along the four directions of up, down, left, and right with a preset step size. Calculate the similarity between the sub-board to be matched after multiple sliding windows and the master board to be matched. Denote the sub-board to be matched after the sliding window with the smallest similarity to the master board to be matched as the feature map to be matched.

[0076] S4b: Repeat step S3b until the remaining feature maps in the adjacent feature map sequence are traversed, then proceed to step S5b.

[0077] S6b: Superimpose all feature maps to be matched with the corresponding feature values ​​of the master image to be matched to generate a fused dense feature map.

[0078] Ultra-high definition fused image generation: The fused dense feature map is subjected to cross-validation of adjacent pixels, elimination of redundant information, removal of outliers, stitching of blind spots, mapping using a global color model, and vignetting correction to obtain an ultra-high definition fused image.

[0079] S1-4. The feature maps are fused at the image feature level using a fusion rule network to obtain an ultra-high-definition fused image.

[0080] Fusion rule networks use multiple relatively accurate edge, contour, and texture feature maps to perform image fusion based on image features.

[0081] This method first sorts multiple feature maps according to the principle of minimizing the sum of the feature values ​​of adjacent feature maps to obtain a sequence of adjacent feature maps; then, it performs feature point matching and superposition processing on multiple feature maps to obtain a fused dense feature map; finally, it performs cross-validation of adjacent pixels, eliminates redundant information, removes outliers, stitches up blind spots, uses a global color model for mapping, and performs vignetting correction on the fused dense feature map to obtain an ultra-high-definition fused image.

[0082] High-resolution, ultra-high-definition fused images are generated from multi-source SAR images using a neural network-based multi-source image fusion algorithm.

[0083] 2. Deep Neural Network Recognition Algorithm

[0084] Figure 2 This diagram illustrates the deep neural network target recognition method based on multi-source SAR image fusion proposed in this invention. The deep neural network recognition model includes an image feature extraction network, a small-scale target extraction network, a region proposal network (RPN), multi-scale region of interest pooling layers, fully connected layers, and a target recognition output layer.

[0085] An image feature extraction network is used to extract edge, contour, and texture image features from the generated ultra-high-definition fused image to obtain a target recognition feature map; the ultra-high-definition fused image is uniformly a 1024*1024 pixel image; the image feature extraction network can be a ResNet-50 network.

[0086] The small-scale target extraction network further extracts small-scale target image features from the target recognition feature map, resulting in image feature maps including small-scale target edges, contours, and textures; the small-scale target extraction network can be a Feature Pyramid Network (FPN).

[0087] The Region Proposal Network (RPN) is used to generate 2D bounding boxes for multiple regions of interest on the image feature map, and to use these generated 2D bounding boxes to segment the sub-image feature map within the corresponding 2D bounding boxes on the image feature map.

[0088] The feature maps of different sizes of sub-images segmented from the 2D bounding box are uniformly generated into a fixed-size image feature map through region of interest pooling (ROIPooling); in a specific embodiment of the present invention, a 7*7 size feature map is uniformly generated.

[0089] The fixed-size image feature maps generated uniformly after pooling are shared to fully connected layers for SAR image target classification and 2D bounding box regression, thus completing the target recognition of the SAR image. The recognition results are output through the target recognition output layer.

[0090] This invention verifies the SAR image fusion and target recognition performance of the deep neural network target recognition method based on multi-source SAR image fusion through SAR image target recognition experiments, and obtains excellent image fusion and target recognition results.

[0091] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A deep neural network target recognition method based on multi-source SAR image fusion, characterized in that The method comprises the following steps: S1, fusing multi-source SAR images to make up for the image information that cannot be shot by a single SAR image, and obtaining a super-definition fused image; S2, using a deep neural network recognition model to perform SAR image target recognition on the super-definition fused image; The multi-source SAR images comprise multi-direction, multi-azimuth and multi-scale SAR images, the multi-direction SAR images are SAR images shot by different cameras from different angles, the multi-azimuth SAR images are SAR images shot by the same camera in different postures, and the multi-scale SAR images are SAR images of different pixel sizes shot by the same camera at different distances while keeping the same posture; The specific method of step S1 is as follows: S1-1, obtaining multi-source SAR images, and performing coordinate correction on multi-direction and multi-scale SAR images shot of the same region to unify them to the same coordinate system; S1-2, uniformly adjusting the multi-direction, multi-azimuth and multi-scale SAR images into the same pixel size of the SAR images to be fused; S1-3, extracting image features in the SAR images to be fused through a convolutional neural network (CNN) to obtain a feature map of each SAR image to be fused; the feature map comprises shape features and texture features of a target; S1-4, performing image feature level fusion on the feature maps through a fusion rule network to obtain a super-definition fused image; The image feature level fusion comprises three steps of adjacent image sorting, key feature point matching and super-definition fused image generation: Adjacent image sorting: sorting the feature maps according to the principle that the sum of the distances of the feature values of adjacent feature maps is the smallest to obtain a sequence of adjacent feature maps; Key feature point matching: performing feature point matching and superimposition processing on the feature maps to obtain a fused dense feature map; Super-definition fused image generation: performing adjacent pixel point cross verification, eliminating redundant information, removing outliers, stitching blind areas, mapping through a global color model and dark corner correction processing on the fused dense feature map to obtain a super-definition fused image; The method of adjacent image sorting is as follows: S1a, randomly selecting a feature map as a reference feature map, and adding the reference feature map to the sequence of adjacent feature maps; S2a, recording the feature maps other than the sequence of adjacent feature maps as candidate feature maps; S3a, determining the candidate feature map with the smallest similarity to the reference feature map as the adjacent image of the reference feature map; S4a, sequentially sorting according to the similarity from small to large as the basis for adjacent image sorting, and adding the adjacent image of the reference feature map to the sequence of adjacent feature maps; S5a, selecting the adjacent image of the reference feature map as a new reference feature map, and repeating steps S2a to S5a until all the feature maps are traversed; The key feature point matching step is as follows: S1b, extracting the feature map with the maximum absolute value of the cumulative sum of all pixel feature values in the sequence of adjacent feature maps as a reference feature map; S2b, extracting key pixel points in the reference feature map, and setting the feature values of the remaining pixel points to zero to obtain a matching master version; S3b, another feature map in the adjacent feature map sequence is extracted, denoted as a to-be-matched sub-board, the to-be-matched sub-board is windowed in four directions of up, down, left and right respectively according to a preset step size, the similarity of the to-be-matched sub-board after windowing and the to-be-matched master board is calculated respectively, and the to-be-matched sub-board after windowing with the smallest similarity with the to-be-matched master board is denoted as a to-be-matched feature map; S4b, step S3b is re-executed until the remaining feature maps in the adjacent feature map sequence are traversed, and step S5b is entered; S6b, all to-be-matched feature maps and feature values of the to-be-matched master board are superimposed together to generate a fusion dense feature map. 2.The method of claim 1, wherein The similarity calculation method of two feature maps is as follows: The feature values of the two feature maps are subtracted and the absolute value is obtained, the distance of the feature values in the two feature maps is obtained, and the sum of the distances of the feature values in the two feature maps is the similarity of the two feature maps. 3.The method of claim 1, wherein The deep neural network recognition model comprises an image feature extraction network, a small-scale target extraction network, a region proposal network, a multi-scale region of interest pooling layer, a connection layer and a target recognition output layer; The image feature extraction network is used for performing edge, contour and texture image feature extraction on the generated fusion image to obtain a target recognition feature map; The small-scale target extraction network further performs small-scale target image feature extraction on the target recognition feature map to obtain an image feature map comprising small-scale target edges, contours and textures; The region proposal network is used for generating a plurality of 2D bounding boxes of regions of interest on the image feature map, and using the generated 2D bounding boxes to respectively segment out sub-image feature maps within the corresponding 2D bounding boxes on the image feature map; Each different size sub-image feature map segmented by the 2D bounding box is uniformly generated into a fixed size image feature map through region of interest pooling; The fixed size image feature map uniformly generated after pooling is shared to the fully connected layer for SAR image target classification and 2D bounding box regression, and the target recognition of the SAR image is completed, and the recognition result is output through the target recognition output layer.

4. The method of claim 3, wherein the method further comprises: The image feature extraction network selects a ResNet-50 network. 5.The method of claim 3, wherein The small-scale target extraction network selects a feature pyramid network.

Images