Semantic segmentation method, device, equipment and storage medium for SAR images

Abstract

The present invention provides a semantic segmentation method, device, equipment and storage medium for SAR images, which can be applied to the field of satellite remote sensing technology. The method comprises: constructing a semantic segmentation model; using the semantic segmentation model, extracting features of different scales of the input image, wherein the features include high-resolution spatial features and low-resolution semantic features, deeply processing and fusing the high-resolution spatial features and low-resolution semantic features to obtain a feature map containing semantic information, and based on the feature map containing semantic information, obtaining a classification result for each pixel in the input image. An effective solution is provided for improving the performance of semantic segmentation tasks.

Description

Technical Field

[0001] The present invention relates to the field of optical communications, and in particular to a semantic segmentation method, device, equipment and storage medium for SAR images. Background Art

[0002] Synthetic aperture radar (SAR) is an active imaging radar that obtains surface information by analyzing and processing echo signals. Compared with passive imaging optical remote sensing, SAR can work all day and all weather, and has the advantages of interferometric measurement capability, multi-polarization, and multi-angle. These advantages make SAR widely used in urban planning, agricultural production, disaster monitoring, military intelligence and other fields.

[0003] Compared with optical images, SAR images are often accompanied by a large amount of noise and geometric deformation, which makes the interpretation of SAR images relatively difficult, especially for buildings. First, the limited resolution of SAR images limits the clear capture of subtle features of buildings, resulting in blurred edges and morphological distortion of buildings, making it difficult to clearly distinguish them from the surrounding environment. Secondly, the diversity of geometric shapes of buildings leads to different radar echoes at different viewing angles, which in turn increases the complexity of identification and classification. In addition, the different materials used in buildings have different reflection properties for SAR signals, which makes the same building present diverse reflection characteristics in different images, increasing the difficulty of interpretation. As an important technology in the field of computer vision, semantic segmentation aims to classify each pixel in the image and divide it into specific categories. Applying semantic segmentation technology to SAR images can significantly reduce the difficulty of SAR image interpretation.

[0004] Due to its unique imaging mechanism, SAR is not restricted by weather, time and light. However, SAR images do not have the detailed information such as color and texture that optical images have, so there are challenges in identifying the fine structure of objects. Compared with optical images, the segmentation performance will be greatly reduced when using SAR images to train and test existing models. The current methods for semantic segmentation of SAR images are still insufficient in segmentation accuracy and inference speed. First, the current methods mainly improve the segmentation performance of the network by adding attention mechanisms and image pyramid pooling operations on the basis of existing models. Although these two approaches have a certain improvement in improving segmentation performance, these operations require a lot of calculations and memory consumption, resulting in a decrease in inference speed. Secondly, the noise problem in SAR images is not considered during feature extraction, and the extracted features are easily disturbed by noise, which affects the subsequent processing results and causes the final segmentation performance to decrease. Finally, the use of spatial information of the input image is neglected, and the fusion processing of semantic information and spatial information is lacking. It is difficult to achieve satisfactory performance by relying solely on semantic information for segmentation. Summary of the invention

[0005] In view of the above problems, the present invention provides a semantic segmentation method, apparatus, device and storage medium for SAR images.

[0006] According to a first aspect of the present invention, a semantic segmentation method for SAR images is provided, comprising:

[0007] Constructing a semantic segmentation model, wherein the semantic segmentation model includes a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module and a segmentation head;

[0008] Using the feature extractor, extracting features of different scales of the input image, the features including high-resolution spatial features and low-resolution semantic features;

[0009] Using the feature reprocessing module and the multi-scale feature deep fusion module, the high-resolution spatial features and the low-resolution semantic features are deeply processed and fused to obtain a feature map containing semantic information;

[0010] The segmentation head is used to obtain a classification result for each pixel in the input image based on the feature map containing semantic information.

[0011] In one embodiment of the present invention, the semantic segmentation model further includes an auxiliary branch, and the method further includes:

[0012] The auxiliary branch is used to extract features of a target area of ​​the input image based on the input image and the mask image, wherein the mask image is generated based on a real annotation image of the input image.

[0013] In one embodiment of the present invention, extracting features of different scales of the input image by using the feature extractor includes:

[0014] Constructing a feature extractor, the feature extractor is a ResNet18 network, the ResNet18 network includes a stem layer and four stages connected in sequence, the stem layer includes three 3×3 convolutions, each of the convolutions is connected with batch normalization and a nonlinear activation function, each of the stages includes two residual blocks, the convolutions of the third and fourth stages of the four stages are dilated convolutions, the dilation rate of the dilated convolution of the third stage is 2, and the dilation rate of the dilated convolution of the fourth stage is 4;

[0015] The input image is input into the ResNet18 network to extract features of different scales of the input image.

[0016] In one embodiment of the present invention, the stem layer is used to perform convolution and pooling operations on the input image to output a first image, wherein the resolution of the first image is one quarter of the resolution of the input image;

[0017] The first stage of the four stages is used to receive the first image and output a first feature map, where the resolution of the first feature map is the same as the resolution of the first image;

[0018] The second stage of the four stages is used to receive the first feature map and output a second feature map, wherein the resolution of the second feature map is one eighth of the resolution of the input image;

[0019] The third stage of the four stages is used to receive the second feature map and output a third feature map, wherein the resolution of the third feature map is one sixteenth of the resolution of the input image;

[0020] The fourth stage of the four stages is used to receive the third feature map and output a fourth feature map, wherein a resolution of the fourth feature map is one thirty-second of a resolution of the input image.

[0021] In one embodiment of the present invention, the deep processing and fusion of the high-resolution spatial features and the low-resolution semantic features to obtain a feature map containing semantic information includes:

[0022] Using the feature reprocessing module, perform 3×3 convolution, batch normalization, and nonlinear activation operations on the second feature map to obtain a second intermediate feature map, where the size of the second intermediate feature map is the same as that of the third feature map, and the step size of the convolution is 2;

[0023] Using the feature reprocessing module, adding the second intermediate feature map and the third feature map to obtain a third intermediate feature map;

[0024] Using the feature reprocessing module, perform 3×3 convolution, batch normalization, and nonlinear activation operations on the third intermediate feature map to obtain an operation result, and add the operation result to the fourth feature map to obtain a fourth intermediate feature map;

[0025] Using the feature reprocessing module, performing 1×1 convolution on the second intermediate feature map, the third intermediate feature map, and the fourth intermediate feature map, respectively, to obtain a second output feature map, a third output feature map, and a fourth output feature map;

[0026] The multi-scale feature deep fusion module is used to obtain the feature map containing semantic information based on the second output feature map, the third output feature map and the fourth output feature map.

[0027] In an embodiment of the present invention, the obtaining of the feature map containing semantic information based on the second output feature map, the third output feature map and the fourth output feature map by using the multi-scale feature deep fusion module includes:

[0028] Using 3×3 convolution to process the second output feature map, the third output feature map, and the fourth output feature map respectively, to obtain a second processing result, a third processing result, and a fourth processing result;

[0029] Upsampling the fourth processing result to obtain an upsampling result;

[0030] Connecting the second processing result, the third processing result and the upsampling result in the channel dimension to obtain a connection result;

[0031] The connection result is channel-compressed using 1×1 convolution to obtain the feature map containing semantic information.

[0032] The obtaining of a classification result for each pixel in the input image by using the segmentation head based on the feature map containing semantic information includes:

[0033] In one embodiment of the present invention, the feature map containing the semantic information is added to the first feature map to obtain an output feature map;

[0034] The output feature map is segmented using the segmentation head to obtain a classification result for each pixel in the input image.

[0035] A second aspect of the present invention provides a semantic segmentation device for SAR images, comprising:

[0036] A model building module, used to build a semantic segmentation model, wherein the semantic segmentation model includes a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module and a segmentation head;

[0037] A feature extraction module, used to extract features of different scales of an input image using the feature extractor, wherein the features include high-resolution spatial features and low-resolution semantic features;

[0038] A feature processing module, used to perform deep processing and fusion of the high-resolution spatial features and low-resolution semantic features using the feature reprocessing module and the multi-scale feature deep fusion module to obtain a feature map containing semantic information;

[0039] The segmentation module is used to obtain a classification result for each pixel in the input image based on the feature map containing semantic information by using the segmentation head.

[0040] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0041] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0042] The semantic segmentation method, device, equipment and medium for SAR images provided by the present invention have at least the following technical effects:

[0043] (1) Aiming at the characteristics of SAR images, the present invention introduces dilated convolution and improves the Stem layer in the encoder stage to propose a feature extraction network with a large receptive field, which can avoid network redundancy while extracting spatial and semantic information. In addition, the network has stronger nonlinear expression ability and can significantly improve the ability to extract high-level features.

[0044] (2) The present invention designs a feature reprocessing module at the decoder stage, which combines multi-scale feature information to deeply extract semantic information, solving the problem of insufficient feature extraction.

[0045] (3) The present invention designs a fast and efficient multi-scale feature deep fusion module to fuse feature maps of different scales. This module uses multi-scale context features to enhance the ability to extract local features and improve the network's ability to recognize small targets.

[0046] (4) The present invention designs an auxiliary branch based on the ground feature information in the data set, constraining the feature extractor to extract features from the area of ​​interest, thereby solving the problem of misjudgment of non-building areas and improving the segmentation performance of building areas.

[0047] (5) After training, compared with the current existing technology, the present invention achieves the best performance in all indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a semantic segmentation method for SAR images according to an embodiment of the present invention is schematically shown;

[0050] Figure 2The overall architecture diagram of the semantic segmentation model according to an embodiment of the present invention is schematically shown;

[0051] Figure 3 A schematic diagram of a feature extractor according to an embodiment of the present invention is shown;

[0052] Figure 4 A schematic diagram of a structure of a Stem layer according to an embodiment of the present invention is shown;

[0053] Figure 5 A diagram schematically showing a semantic information extraction diagram of a feature reprocessing module according to an embodiment of the present invention;

[0054] Figure 6 A feature extraction diagram of a multi-scale feature deep fusion module according to an embodiment of the present invention is schematically shown;

[0055] Figure 7 A schematic diagram of feature map segmentation according to an embodiment of the present invention is schematically shown;

[0056] Figure 8 A schematic diagram of an auxiliary branch structure according to an embodiment of the present invention is shown;

[0057] Fig. 9 The structure block diagram of the semantic segmentation device for SAR images according to an embodiment of the present invention is schematically shown;

[0058] Fig.10 A block diagram of an electronic device suitable for a semantic segmentation method for SAR images according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0059] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0060] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0061] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0062] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0063] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision, invention and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0064] In the technical solution of the present invention, the acquisition, collection, storage, use, processing, transmission, provision, invention and application of data all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0065] At present, the methods for semantic segmentation of SAR images are mainly divided into two categories:

[0066] 1. Traditional methods

[0067] Traditional semantic segmentation methods usually use manually designed feature extraction methods. The extracted features may be associated with specific SAR image characteristics, making the results more interpretable. Traditional segmentation methods mainly include: segmentation methods based on edge detection, segmentation methods based on clustering theory, and segmentation methods based on random field models.

[0068] The segmentation method based on edge detection attempts to identify obvious boundaries or edges in the image, and then classifies the pixels within the boundaries to achieve image segmentation. In SAR images, edges usually represent the boundaries between different objects or scenes, so the use of edge detection for segmentation has a certain effect. Representative methods include the Canny algorithm. The segmentation method based on clustering theory segments the image by combining pixels with similar features in the image into a group. Common clustering algorithms include K-means and fuzzy C-means clustering algorithms. The K-Means algorithm divides pixels into K clusters, where each cluster represents a specific type of area. The algorithm iteratively assigns pixels to the clusters closest to them and updates the center of the cluster until convergence is reached. Fuzzy C-Means is an extension of K-Means, which allows a pixel to belong to multiple clusters instead of being rigidly assigned to one cluster. The segmentation method based on the random field model mainly segments the image by the spatial relationship between pixels and the correlation between features. Representative models include the Markov random field model and the conditional random field model. The MRF model regards the pixels in the image as random variables and establishes a probability model based on the spatial relationship and similarity between pixels. The CRF model is an extension of the MRF, which considers modeling random fields given the observed data. In SAR image segmentation, CRF can combine the intensity information, texture features and spatial relationships of pixels to model and obtain more accurate segmentation results.

[0069] 2. Deep Learning Methods

[0070] With the rapid increase in data volume in recent years, deep learning, as a method with large-scale data processing capabilities, has shown great potential in the field of remote sensing image segmentation. Compared with traditional methods, deep learning can automatically learn and extract complex features without relying on a lot of prior knowledge and manual feature extraction.

[0071] As an end-to-end deep learning method, the fully convolutional network (FCN) allows input of images of any size and performs pixel-level predictions through convolutional layers and upsampling layers to achieve semantic segmentation. The U-Net structure is a variant of the FCN structure. It adopts a symmetrical network structure. The overall structure can be divided into two parts. The upper part is used for feature extraction, and the lower part is used for upsampling to restore to the input resolution, and skip connections are used to retain shallow edge features. The DeepLab series of networks use hollow space pyramid pooling to capture features of different scales to improve the accuracy of segmentation and detail retention, showing excellent segmentation performance. With the continuous development of deep learning, fast inference networks have gradually become a research hotspot. BiSeNet decomposes the segmentation task into a fast low-resolution branch and a slow high-resolution branch, and then fuses information at different levels to achieve a balance between speed and accuracy. STDCNet uses fewer parameters for feature extraction through the STDC module, and adds Detail Guidance to improve the segmentation performance of the network.

[0072] In view of the problems existing in the existing models, the present invention proposes a semantic segmentation method for SAR images. The constructed semantic segmentation model mainly consists of four parts: a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module, a segmentation head and an auxiliary branch. In view of the problem of redundancy in the current network structure, only one branch is used in the feature extractor to downsample the image, extract the low-resolution semantic features and high-resolution spatial information of the image, and avoid structural redundancy. At the same time, in order to ensure that the inference speed is not affected, a shallower network is set in the feature extraction stage. In view of the problem of small receptive field, some convolution operations in the feature extractor are replaced by dilated convolution. Through this design, the increase of the receptive field is achieved without increasing the amount of calculation, and the receptive field can be adjusted by changing the parameters to achieve the best performance. In view of the problem of lack of information reprocessing, a feature reprocessing module and a multi-scale feature fusion module are designed to deeply process and fuse the multi-scale information extracted from the feature extractor. The fused multi-scale features can provide more comprehensive and richer semantic information, which helps to improve the model's ability to understand the scene. Finally, in order to reduce the misjudgment of objects in non-target areas, an auxiliary branch is designed to constrain the feature extractor to extract features from the area of ​​interest. It should be noted that the design of the present invention has significant advantages in improving the model's accurate capture and feature extraction of key areas, and provides an effective solution for improving the performance of semantic segmentation tasks.

[0073] Figure 1 The flowchart of the semantic segmentation method for SAR images according to an embodiment of the present invention is schematically shown.

[0074] like Figure 1As shown, the semantic segmentation method for SAR images of this embodiment includes operations S110 to S130.

[0075] In operation S110, a semantic segmentation model is constructed, where the semantic segmentation model includes a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module, and a segmentation head.

[0076] like Figure 2 As shown, the overall architecture of the semantic segmentation model constructed by the semantic segmentation method for SAR images provided by the present invention is based on an encoder-decoder structure. The encoder includes a feature extractor, and the decoder includes a feature reprocessing module, a multi-scale deep fusion module and a segmentation head. In the feature extractor part, the features of the input image at different scales are extracted by the ResNet18 network, including high-resolution spatial features and low-resolution semantic features. In order to meet the requirements of fast reasoning, the present invention sets fewer convolutional layers in the encoder, which causes the extracted semantic information to be in an under-processed state. Therefore, the present invention designs a feature reprocessing module and a multi-scale feature deep fusion module in the decoder part to further process the under-processed semantic information. Although the processed semantic feature map contains rich semantic information, multiple downsamplings lead to the loss of its spatial information. Generally speaking, the shallow feature resolution is high and rich in a large amount of spatial information. In order to make up for the spatial information in the feature map, the present invention fuses the shallow features in the encoder with the processed semantic features by direct addition. Finally, the feature map is sent to the segmentation head to complete the classification of each pixel. In addition, the feature map extracted by the encoder will also be sent to the auxiliary branch. In the auxiliary branch, the input feature map and mask map calculate the loss function and back-propagate the gradient to reduce the misjudgment of non-building parts and improve the quality of the predicted image.

[0077] In operation S120, the feature extractor is used to extract features of different scales of the input image, where the features include high-resolution spatial features and low-resolution semantic features.

[0078] In one embodiment of the present invention, a feature extractor may be used to implement operation S110 to extract features of different scales of the input image. The feature extractor may use a ResNet structure including a residual block, such as Figure 3 As shown. In the embodiment of the present invention, the residual connections in these residual blocks allow the network to propagate information across layers, thereby solving the gradient vanishing and information loss problems that may occur when training very deep networks. This allows the network to be designed to be deeper and wider, thereby learning more abstract and complex features, and improving the effect of training deep models.

[0079] In one embodiment of the present invention, operation S120 extracts features of different scales of the input image, including: constructing a feature extractor, which is a ResNet18 network, the ResNet18 network including a stem layer and four stages Stage1-4 connected in sequence, the stem layer including three 3×3 convolutions, each of which is connected with batch normalization and a nonlinear activation function after the convolution, each of which includes two residual blocks, and the convolutions of the third stage Stage3 and the fourth stage Stage4 of the four stages are dilated convolutions, the dilation rate of the dilated convolution of the third stage Stage3 is 2, and the dilation rate of the dilated convolution of the fourth stage Stage4 is 4; the input image is input into the ResNet18 network to extract features of different scales of the input image.

[0080] According to this embodiment, the 7×7 convolution in the Stem layer is replaced by three 3×3 convolutions, and batch normalization and nonlinear activation functions are added after each convolution, such as Figure 4 As shown. Although the receptive fields of the two are the same, the latter has a significantly higher degree of nonlinearity and can represent more complex functions. In addition, the embodiment of the present invention replaces the convolution in Stage 3 and Stage 4 with dilated convolution, with dilation rates of 2 and 4, respectively. This change increases the receptive field of the network, which is beneficial to the extraction of context information.

[0081] In one embodiment of the present invention, the stem layer is used to perform convolution and pooling operations on the input image, and output a first image, the resolution of which is one quarter of the resolution of the input image; the first stage of the four stages, Stage 1, is used to receive the first image and output a first feature map , the first feature map The resolution of the first image is the same as the resolution of the first image; the second stage Stage2 of the four stages is used to receive the first feature map , output the second feature map , the second feature map The resolution is one eighth of the resolution of the input image; the third stage of the four stages, Stage 3, is used to receive the second feature map , output the third feature map , the third characteristic map The resolution is one sixteenth of the resolution of the input image; the fourth stage Stage4 of the four stages is used to receive the third feature map , output the fourth feature map , the fourth characteristic map The resolution is one thirty-second of the resolution of the input image.

[0082] In the embodiment of the present invention, the input image is first processed by the Stem layer in the feature extractor. The Stem layer mainly performs convolution and pooling operations, and the output image will become a quarter of the original resolution. Next, these images will pass through the four stages of the ResNet18 network, each of which consists of two Basic Blocks. Except for Stage 1, no downsampling is performed in the other three stages Stage 2-4. Therefore, the first feature map output in each stage , the second feature map , the third feature map , the fourth characteristic map The resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively.

[0083] In operation S130, the feature reprocessing module and the multi-scale feature deep fusion module are used to deeply process and fuse the high-resolution spatial features and the low-resolution semantic features to obtain a feature map containing semantic information.

[0084] In one embodiment of the present invention, a feature reprocessing module and a multi-scale feature deep fusion module can be used to implement operation S120 to deeply process and fuse the high-resolution spatial features and the low-resolution semantic features to obtain a feature map containing semantic information.

[0085] In the embodiment of the present invention, after the input image passes through the feature extractor, feature maps of multiple scales (the first feature map , the second feature map , the third feature map , the fourth characteristic map ). Each pixel of the feature map corresponds to a local area in the input image. The low-resolution feature map undergoes more convolution and downsampling, resulting in a reduction in spatial dimension, but an increase in the receptive field. This large receptive field allows each pixel to represent a wider range of semantic information, which can provide the network with more comprehensive contextual information and help better understand the objects and scenes in the image. The present invention further extracts semantic information through a feature reprocessing module combined with feature maps of multiple scales, such as Figure 5 shown.

[0086] In one embodiment of the present invention, the following operations are implemented through a feature reprocessing module: performing 3×3 convolution, batch normalization, and nonlinear activation operations on the second feature map to obtain a second intermediate feature map, the size of the second intermediate feature map is the same as the size of the third feature map, and the step size of the convolution is 2; adding the second intermediate feature map and the third feature map to obtain a third intermediate feature map; performing 3×3 convolution, batch normalization, and nonlinear activation operations on the third intermediate feature map to obtain an operation result, and adding the operation result to the fourth feature map to obtain a fourth intermediate feature map; performing 1×1 convolution on the second intermediate feature map, the third intermediate feature map, and the fourth intermediate feature map, respectively, to obtain a second output feature map , the third output feature map And the fourth output feature map .

[0087] In the feature reprocessing module, , , As input. First Perform 3×3 convolution, batch normalization and nonlinear activation operations, and the stride of the convolution is set to 2. Through this series of operations, Completed downsampling and channel expansion, and have the same size. and Then the addition result is completed with After the same operation, Finally, the outputs of the three stages are compressed by 1×1 convolution, reducing the number of channels to half of the original. The processing process can be expressed as follows:

[0088]

[0089] in, represents 1×1 convolution, batch normalization, and ReLu activation operations, Represents 3×3 convolution, batch normalization and ReLu activation operations, the second output feature map , the third output feature map , the fourth output feature map Represents output feature maps of different scales.

[0090] In one embodiment of the present invention, the following operations are implemented through a multi-scale feature deep fusion module: the second output feature map, the third output feature map and the fourth output feature map are respectively processed using 3×3 convolution to obtain a second processing result, a third processing result and a fourth processing result; the fourth processing result is upsampled to obtain an upsampled result; the second processing result, the third processing result and the upsampled result are connected in the channel dimension to obtain a connection result; the connection result is channel compressed using 1×1 convolution to obtain the feature map containing semantic information.

[0091] The context information of different scales of the image is obtained through the feature reprocessing module, but the feature map of each scale can only provide information at this scale. In remote sensing images, different objects often have different scales, which leads to the fact that the context information of a single scale cannot fully describe all objects. In order to solve this problem, related methods have proposed pyramid pooling modules and void space pyramid pooling. However, these operations require a lot of calculations and memory consumption, resulting in a low inference speed. In view of the above situation, the present invention proposes the following Figure 6 The multi-scale feature deep fusion module shown.

[0092] In the multi-scale feature deep fusion module, , , As input, we first use 3×3 convolution to process the feature maps of each scale separately. It should be noted that The stride of the convolution is set to 2 to achieve In addition, Upsampling is performed for subsequent fusion. Finally, all feature maps are connected in the channel dimension and channel compression is performed using 1×1 convolution. The processing process can be expressed as follows

[0093]

[0094] in, Represents channel dimension connection, Represents an upsampling operation.

[0095] In operation S140, the segmentation head is used to obtain a classification result for each pixel in the input image based on the feature map containing semantic information.

[0096] After a series of processing, a feature map containing semantic information is obtained. However, for semantic segmentation, it is difficult to achieve satisfactory performance only with semantic information due to the lack of details in the original image. Shallow high-resolution features are less downsampled and can more finely represent the details and structure of the original image. Therefore, the embodiment of the present invention uses the feature map containing semantic information and the first feature map in the feature extractor Add to enrich the details in the feature map and obtain the output feature map. Finally, Figure 7 As shown, the segmentation head is used to segment the output feature map and classify each pixel of the image.

[0097] In one embodiment of the present invention, Figure 2 The method shown also includes: using the auxiliary branch to extract features of the target area of ​​the input image based on the input image and the mask image, and the mask image is generated based on the real annotation image of the input image.

[0098] In order to reduce the misjudgment of pixels in non-building areas in the segmentation results, the present invention uses the positional relationship between the object categories in the data set to design an auxiliary branch. In the auxiliary branch, we classify the pixels related to buildings in the real annotation image into one category and generate a mask image. Then, the feature map in the feature extractor is used to extract the building pixels. Segmentation is performed and the loss is calculated with the mask image. This design allows the feature extractor to segment the target area and reduce the misjudgment of pixels in non-target areas. The structure of the auxiliary branch is as follows: Figure 8 shown.

[0099] The embodiment of the present invention selects the cross entropy loss function (loss_CE), the Lovasz loss function (loss_Lo) and the Dice loss function (loss_DL). The expressions are as follows:

[0100]

[0101]

[0102]

[0103] Among them, N represents the total number of pixels, i represents the pixel index, and M represents the number of pixel categories. represents the symbolic function, represents the error vector, R represents the segmentation result, and G represents the mask image.

[0104] In one embodiment of the present invention, the SARMV3D-BIS dataset can be used to train the model. The original images of the dataset are 14 SAR images of the Omaha area in the United States. The dataset contains 234 SAR images of 1024×1024 size, including more than 6,000 buildings. Compared with other datasets, this dataset can provide more detailed annotations, and each building is annotated with its facade, roof and shadow part.

[0105] In one embodiment of the present invention, Figure 2 The method also includes: using average intersection-over-union ratio, average accuracy and overall accuracy as indicators for evaluating the classification result.

[0106] This paper uses the widely recognized mean intersection over union (mIoU), average accuracy (mAcc) and overall accuracy (aAcc) as indicators for evaluating segmentation quality, and comprehensively evaluates the method. For inference speed, the number of frames processed per second (FPS) is used as the evaluation indicator.

[0107] The expression of average intersection-over-union ratio is as follows:

[0108]

[0109] Among them, i represents the pixel category, n represents the number of image categories, GT represents the true annotation, and Pred represents the segmentation result.

[0110] The expressions for average accuracy (mAcc) and overall accuracy (aAcc) are as follows:

[0111]

[0112] Among them, TP, FP, FN, and TN represent the true positive class, false positive class, false negative class, and true negative class in the confusion matrix, respectively.

[0113] Based on the above semantic segmentation method for SAR images, the present invention also provides a semantic segmentation device for SAR images. Fig. 9 The device is described in detail.

[0114] Fig. 9 The structural block diagram of a semantic segmentation device for SAR images according to an embodiment of the present invention is schematically shown.

[0115] like Fig. 9 As shown, the semantic segmentation device 900 for SAR images of this embodiment includes a model building module 910 , a feature extraction module 920 , a feature processing module 930 , and a segmentation module 940 .

[0116] The model building module 910 is used to build a semantic segmentation model, which includes a feature extractor, a feature reprocessing module, a multi-scale feature depth fusion module and a segmentation head. In one embodiment, the model building module 910 can be used to perform the operation S110 described above, which will not be repeated here.

[0117] The feature extraction module 920 is used to use the feature extractor to extract features of different scales of the input image, the features including high-resolution spatial features and low-resolution semantic features. In one embodiment, the feature extraction module 920 can be used to perform the operation S120 described above, which will not be repeated here.

[0118] The feature processing module 930 is used to use the feature reprocessing module and the multi-scale feature deep fusion module to perform deep processing and fusion on the high-resolution spatial features and low-resolution semantic features to obtain a feature map containing semantic information. In one embodiment, the feature processing module 930 can be used to perform the operation S130 described above, which will not be repeated here.

[0119] The segmentation module 940 is used to obtain a classification result for each pixel in the input image using the segmentation head based on the feature map containing semantic information. In one embodiment, the segmentation module 940 can be used to perform the operation S140 described above, which will not be described in detail here.

[0120] According to an embodiment of the present invention, the semantic segmentation model also includes an auxiliary branch, and the device 900 also includes: an auxiliary module, used to use the auxiliary branch to extract features of the target area of ​​the input image based on the input image and the mask image, and the mask image is generated based on the real annotation image of the input image.

[0121] According to an embodiment of the present invention, using the feature extractor to extract features of different scales of the input image includes:

[0122] Construct a feature extractor, which is a ResNet18 network. The ResNet18 network includes a stem layer and four stages connected in sequence. The stem layer includes three 3×3 convolutions. Each of the convolutions is connected with batch normalization and a nonlinear activation function. Each of the stages includes two residual blocks. The convolutions of the third and fourth stages of the four stages are dilated convolutions. The dilation rate of the dilated convolution of the third stage is 2, and the dilation rate of the dilated convolution of the fourth stage is 4.

[0123] The input image is input into the ResNet18 network to extract features of different scales of the input image.

[0124] According to an embodiment of the present invention, the stem layer is used to perform convolution and pooling operations on the input image to output a first image, wherein the resolution of the first image is one quarter of the resolution of the input image;

[0125] The first stage of the four stages is used to receive the first image and output a first feature map, wherein the resolution of the first feature map is the same as the resolution of the first image;

[0126] The second stage of the four stages is used to receive the first feature map and output a second feature map, where the resolution of the second feature map is one eighth of the resolution of the input image;

[0127] The third stage of the four stages is used to receive the second feature map and output a third feature map, wherein the resolution of the third feature map is one sixteenth of the resolution of the input image;

[0128] The fourth stage of the four stages is used to receive the third feature map and output a fourth feature map, wherein a resolution of the fourth feature map is one thirty-second of a resolution of the input image.

[0129] According to an embodiment of the present invention, the high-resolution spatial features and the low-resolution semantic features are deeply processed and fused to obtain a feature map containing semantic information, including:

[0130] Using the feature reprocessing module, perform 3×3 convolution, batch normalization and nonlinear activation operations on the second feature map to obtain a second intermediate feature map, where the size of the second intermediate feature map is the same as that of the third feature map, and the step size of the convolution is 2;

[0131] Using the feature reprocessing module, adding the second intermediate feature map and the third feature map to obtain a third intermediate feature map;

[0132] Using the feature reprocessing module, perform 3×3 convolution, batch normalization, and nonlinear activation operations on the third intermediate feature map to obtain an operation result, and add the operation result to the fourth feature map to obtain a fourth intermediate feature map;

[0133] Using the feature reprocessing module, the second intermediate feature map, the third intermediate feature map, and the fourth intermediate feature map are respectively subjected to 1×1 convolution to obtain a second output feature map, a third output feature map, and a fourth output feature map;

[0134] The multi-scale feature deep fusion module is utilized to obtain the feature map containing semantic information based on the second output feature map, the third output feature map and the fourth output feature map.

[0135] According to an embodiment of the present invention, the multi-scale feature deep fusion module is used to obtain the feature map containing semantic information based on the second output feature map, the third output feature map and the fourth output feature map, including:

[0136] Using 3×3 convolution to process the second output feature map, the third output feature map, and the fourth output feature map, respectively, to obtain a second processing result, a third processing result, and a fourth processing result;

[0137] Upsampling the fourth processing result to obtain an upsampling result;

[0138] Connecting the second processing result, the third processing result and the up-sampling result in a channel dimension to obtain a connection result;

[0139] The connection result is channel compressed using 1×1 convolution to obtain the feature map containing semantic information.

[0140] According to an embodiment of the present invention, the device 900 further includes:

[0141] The evaluation module is used to use the average intersection-over-union ratio, average accuracy and overall accuracy as indicators for evaluating the classification results.

[0142] According to an embodiment of the present invention, any multiple modules among the model building module 910, the feature extraction module 920, the feature processing module 930, and the segmentation module 940 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the model building module 910, the feature extraction module 920, the feature processing module 930, and the segmentation module 940 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the model building module 910, the feature extraction module 920, the feature processing module 930, and the segmentation module 940 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0143] Fig.10 A block diagram of an electronic device suitable for implementing a semantic segmentation method for SAR images according to an embodiment of the present invention is schematically shown.

[0144] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0145] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other through bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0146] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0147] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0148] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0149] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0150] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0151] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0152] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0153] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention and in the specification may be combined in various combinations and / or combinations, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined in various combinations and / or combinations. All of these combinations and / or combinations fall within the scope of the present invention.

[0156] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A semantic segmentation method for SAR images, characterized in that: include: Constructing a semantic segmentation model, the semantic segmentation model includes a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module and a segmentation head; the feature extractor is a ResNet18 network, and the ResNet18 network includes a stem layer and four stages connected in sequence; Inputting an input image into the ResNet18 network, extracting features of the input image at different scales, wherein the features include high-resolution spatial features and low-resolution semantic features; the stem layer is used to perform convolution and pooling operations on the input image, and output a first image; The first stage of the four stages is used to receive the first image and output a first feature map; The second stage of the four stages is used to receive the first feature map and output a second feature map; The third stage of the four stages is used to receive the second feature map and output a third feature map; The fourth stage of the four stages is used to receive the third feature map and output a fourth feature map; Using the feature reprocessing module and the multi-scale feature deep fusion module, the high-resolution spatial features and the low-resolution semantic features are deeply processed and fused to obtain a feature map containing semantic information, including: using the feature reprocessing module, performing 3×3 convolution, batch normalization and non-linear activation operations on the second feature map to obtain a second intermediate feature map; using the feature reprocessing module, adding the second intermediate feature map and the third feature map to obtain a third intermediate feature map; using the feature reprocessing module, performing 3×3 convolution, batch normalization and non-linear activation operations on the third intermediate feature map to obtain an operation result, and adding the operation result to the fourth feature map to obtain a fourth intermediate feature map; using the feature reprocessing module, performing 1×1 convolution on the second intermediate feature map, the third intermediate feature map and the fourth intermediate feature map respectively to obtain a second output feature map, a third output feature map and a fourth output feature map; using the multi-scale feature deep fusion module, based on the second output feature map, the third output feature map and the fourth output feature map, obtaining the feature map containing semantic information; The segmentation head is used to obtain a classification result for each pixel in the input image based on the feature map containing semantic information.

2. The semantic segmentation method for SAR images according to claim 1, characterized in that: The semantic segmentation model also includes an auxiliary branch, and the method further includes: The auxiliary branch is used to extract features of a target area of ​​the input image based on the input image and the mask image, wherein the mask image is generated based on a real annotation image of the input image.

3. The semantic segmentation method for SAR images according to claim 1, characterized in that: The stem layer includes three 3×3 convolutions, each of which is connected with batch normalization and a nonlinear activation function, each of the stages includes two residual blocks, and the convolutions in the third and fourth stages of the four stages are dilated convolutions, the dilation rate of the dilated convolution in the third stage is 2, and the dilation rate of the dilated convolution in the fourth stage is 4.

4. The semantic segmentation method for SAR images according to claim 3, characterized in that: The resolution of the first image is one quarter of the resolution of the input image; The resolution of the first feature map is the same as the resolution of the first image; The resolution of the second feature map is one eighth of the resolution of the input image; The resolution of the third feature map is one sixteenth of the resolution of the input image; A resolution of the fourth feature map is one thirty-second of a resolution of the input image.

5. The semantic segmentation method for SAR images according to claim 4, characterized in that: The size of the second intermediate feature map is the same as that of the third feature map, and the step size of the convolution is 2.

6. The semantic segmentation method for SAR images according to claim 5, characterized in that: The using the multi-scale feature deep fusion module to obtain the feature map containing semantic information based on the second output feature map, the third output feature map, and the fourth output feature map comprises: Using 3×3 convolution to process the second output feature map, the third output feature map, and the fourth output feature map respectively, to obtain a second processing result, a third processing result, and a fourth processing result; Upsampling the fourth processing result to obtain an upsampling result; Connecting the second processing result, the third processing result and the upsampling result in the channel dimension to obtain a connection result; The connection result is channel-compressed using 1×1 convolution to obtain the feature map containing semantic information.

7. The semantic segmentation method for SAR images according to claim 4, characterized in that: The obtaining of a classification result for each pixel in the input image by using the segmentation head based on the feature map containing semantic information includes: Adding the feature map containing the semantic information to the first feature map to obtain an output feature map; The output feature map is segmented using the segmentation head to obtain a classification result for each pixel in the input image.

8. A semantic segmentation device for SAR images, characterized in that: include: A model building module, used to build a semantic segmentation model, the semantic segmentation model includes a feature extractor, a feature reprocessing module, a multi-scale feature deep fusion module and a segmentation head; the feature extractor is a ResNet18 network, and the ResNet18 network includes a stem layer and four stages connected in sequence; A feature extraction module, used to input an input image into the ResNet18 network, extract features of the input image at different scales, wherein the features include high-resolution spatial features and low-resolution semantic features; the stem layer is used to perform convolution and pooling operations on the input image, and output a first image; The first stage of the four stages is used to receive the first image and output a first feature map; The second stage of the four stages is used to receive the first feature map and output a second feature map; The third stage of the four stages is used to receive the second feature map and output a third feature map; the fourth stage of the four stages is used to receive the third feature map and output a fourth feature map; A feature processing module, used to use the feature reprocessing module and the multi-scale feature deep fusion module to perform deep processing and fusion on the high-resolution spatial features and the low-resolution semantic features to obtain a feature map containing semantic information, including: using the feature reprocessing module to perform 3×3 convolution, batch normalization and non-linear activation operations on the second feature map to obtain a second intermediate feature map; using the feature reprocessing module to add the second intermediate feature map and the third feature map to obtain a third intermediate feature map; using the feature reprocessing module to perform 3×3 convolution, batch normalization and non-linear activation operations on the third intermediate feature map to obtain an operation result, and adding the operation result to the fourth feature map to obtain a fourth intermediate feature map; using the feature reprocessing module to perform 1×1 convolution on the second intermediate feature map, the third intermediate feature map and the fourth intermediate feature map respectively to obtain a second output feature map, a third output feature map and a fourth output feature map; using the multi-scale feature deep fusion module to obtain the feature map containing semantic information based on the second output feature map, the third output feature map and the fourth output feature map; The segmentation module is used to obtain a classification result for each pixel in the input image based on the feature map containing semantic information by using the segmentation head.

9. An electronic device, comprising: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images