A semantic segmentation method and device based on edge perception function of SAR image fusion

Abstract

The application discloses a semantic segmentation method and device based on SAR image fusion edge perception function, and relates to the technical field of remote sensing image processing and computer vision. The method comprises the following steps: constructing a semantic segmentation network based on SAR images; extracting features of SAR images based on a CNN feature extraction encoder combined with a hollow convolution to obtain SAR image features; extracting features of edge contours between different subjects in the SAR images through an edge perception module to obtain features of edge contours between different subjects in the SAR images; calculating a semantic segmentation loss function of the network through a true value label and a probability distribution obtained by decoding SAR image features; calculating an edge segmentation loss function of the network through the true value label and a probability distribution obtained by decoding edge contour features; constructing a total loss function of the network according to the two loss functions; and training the total loss function to obtain a trained semantic segmentation network. The application can enhance the semantic segmentation interpretation effect.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, and in particular to a semantic segmentation method and apparatus based on SAR image fusion edge perception function. Background Technology

[0002] Synthetic Aperture Radar (SAR), as an active microwave remote sensing sensor, has the ability to acquire data in all weather conditions and around the clock, and can penetrate clouds and fog, unaffected by lighting conditions. This makes SAR imagery irreplaceable in flood monitoring, water body extraction, and land cover analysis.

[0003] However, the unique coherent imaging mechanism of SAR images results in a large amount of speckle noise, and the texture features of ground objects differ greatly from those in optical images, exhibiting low signal-to-noise ratios and blurred boundaries. When traditional semantic segmentation methods are directly applied to SAR images, they often suffer from the following problems: traditional convolutional neural networks easily lose subtle texture features during downsampling, leading to an imbalance in their ability to capture ground objects at different scales (such as small rivers and large lakes). Simultaneously, due to the inherent boundary ambiguity of SAR images, relying solely on semantic supervision makes it difficult to segment sharp edges, resulting in "jagged" or overly smooth extraction results that fail to meet the demands of high-precision mapping. Furthermore, existing methods have low edge information utilization, mostly using edges only as auxiliary supervision signals without explicitly fusing edge features back into the backbone network to assist in feature recovery. Summary of the Invention

[0004] To address the technical problem of existing technologies easily losing subtle texture features, leading to an imbalance in the ability to capture features at different scales, this invention provides a semantic segmentation method and apparatus based on SAR image fusion edge perception. The technical solution is as follows: On the one hand, a semantic segmentation method based on SAR image fusion edge perception function is provided. This method is implemented by a semantic segmentation device based on SAR image fusion edge perception function, and includes: S1. Obtain SAR images and corresponding ground truth labels; use the Canny edge detection algorithm to extract edge contour images between different subjects in the SAR images; S2. Construct a semantic segmentation network based on SAR images; wherein, the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge-aware module, and a decoder with feature fusion function; S3. Input the SAR image and the edge contour image into the feature extraction encoder for feature extraction to obtain SAR image features and the edge contour features between various subjects in the SAR image. S4. Input the features of the edge contour into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map, construct the segmentation loss function of the edge contour between each subject in the SAR image. S5. Input the features of the edge contour and the SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information; construct a SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label; S6. Based on the semantic segmentation loss function of SAR image and the segmentation loss function of the edge contours between various subjects in the SAR image, construct the total loss function of the semantic segmentation network based on SAR image; train the semantic segmentation network based on SAR image based on the total loss function to obtain the trained semantic segmentation network. S7. Obtain the SAR image to be detected; input the SAR image to be detected into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

[0005] On the other hand, a semantic segmentation device based on SAR image fusion edge sensing function is provided. This device is applied to a semantic segmentation method based on SAR image fusion edge sensing function. The device includes: The image acquisition unit is used to acquire SAR images and corresponding ground truth labels; the Canny edge detection algorithm is used to extract the edge contour images between different subjects in the SAR images; The first building unit is used to build a semantic segmentation network based on SAR images; wherein, the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge awareness module, and a decoder with feature fusion function; The feature extraction unit is used to input the SAR image and the edge contour image into the feature extraction encoder for feature extraction, so as to obtain SAR image features and edge contour features between various subjects in the SAR image. The second construction unit is used to input the features of the edge contour into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; and to construct the segmentation loss function of the edge contour between each subject in the SAR image based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map. The third construction unit is used to input the features of the edge contour and the SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information; and to construct a SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label. The training unit is used to construct the total loss function of the semantic segmentation network based on SAR images based on the semantic segmentation loss function of SAR images and the segmentation loss function of the edge contours between various subjects in SAR images; and to train the semantic segmentation network based on SAR images based on the total loss function to obtain the trained semantic segmentation network. The segmentation unit is used to acquire the SAR image to be detected; the SAR image to be detected is input into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

[0006] On the other hand, a semantic segmentation device based on SAR image fusion edge perception function is provided. The semantic segmentation device based on SAR image fusion edge perception function includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-described semantic segmentation methods based on SAR image fusion edge perception function is implemented.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described semantic segmentation methods based on SAR image fusion edge perception function.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention, by integrating edge perception functionality and introducing a multi-task learning strategy with edge constraints, can fully exploit the edge features of SAR images at different scales and improve the ability to identify complex ground feature boundaries.

[0009] The embodiments of the present invention effectively take into account both local details and global contextual information through multi-scale feature fusion, thereby improving the extraction capability of small targets and large-scale targets.

[0010] The present invention can enhance the binary semantic segmentation and interpretation of various land cover types in SAR images, including but not limited to water body segmentation, building segmentation, road segmentation and other infrastructure segmentation.

[0011] The embodiments of the present invention do not require strict prior assumptions or complex artificial feature design for the input SAR image, and are applicable to SAR data of different resolutions and different imaging modes, with good versatility and scalability. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a semantic segmentation method based on SAR image fusion edge perception function provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of a semantic segmentation structure based on SAR image fusion edge perception function provided in an embodiment of the present invention; Figure 3 This is a block diagram of a semantic segmentation device based on SAR image fusion edge perception function provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a semantic segmentation device based on SAR image fusion edge perception function provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0015] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0016] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0017] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] This invention provides a semantic segmentation method based on SAR image fusion edge perception. This method can be implemented by a semantic segmentation device based on SAR image fusion edge perception, which can be a terminal or a server. Figure 1 The flowchart shown is a semantic segmentation method based on SAR image fusion edge sensing. The processing flow of this method may include the following steps: S1. Obtain SAR images and corresponding ground truth labels; use the Canny edge detection algorithm to extract edge contour images between different subjects within the SAR images.

[0020] Specifically, the Canny edge detection algorithm or the Sobel edge detection algorithm is used to extract the edge contours between different subjects in the SAR image and save them as an image.

[0021] S2. Construct a semantic segmentation network based on SAR images; wherein the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge-aware module, and a decoder with feature fusion function.

[0022] in, Figure 2 This is a schematic diagram of a semantic segmentation structure based on SAR image fusion edge perception function provided in an embodiment of the present invention.

[0023] Optionally, the CNN-based feature extraction encoder that incorporates dilated convolutions adopts a fully embedded dilated spatial pyramid feature extraction architecture to replace the traditional single convolution stacking mode. The feature extraction architecture includes four consecutive downsampling stages; each stage integrates a hollow spatial pyramid pooling module; and each module constructs five parallel feature extraction branches. Among them, the five parallel feature extraction branches include: one regular convolution branch, three dilated convolution branches with different dilation rate parameters, and one image-level context-aware branch containing global average pooling operation. Among them, the conventional convolutional branch is used to preserve the inherent local details of the features; Among them, the dilated convolution branch is used to expand the receptive field through sparse sampling without reducing the resolution.

[0024] Among them, the diverse features extracted from the above five branches are concatenated in the channel dimension and then deeply fused through a projection layer containing convolution, batch normalization and nonlinear activation functions.

[0025] Among them, the edge perception module is used to extract the edge contour features of each subject in the SAR image.

[0026] The initialization layer is used to perform preliminary 2×2 downsampling on the large input image to reduce spatial resolution and subsequent computation, while mapping the input SAR image to 8-32 dimension feature channels.

[0027] S3. Input the SAR image and edge contour image into the feature extraction encoder for feature extraction to obtain SAR image features and edge contour features between various subjects in the SAR image.

[0028] Optionally, the specific implementation process of S3 includes S31-S34: S31. Input SAR images and edge contour images into the initialization layer to extract features and output the basic feature map; S32. Input the basic feature map into the regular convolutional branch and output the local feature map; S32. Input the basic feature map into the three dilated convolution branches respectively to extract contextual information at different scales and output feature maps at different scales. S33. Input the basic feature map into the image-level context-aware branch containing the global average pooling operation to extract global semantic information and output the global feature map. Alternatively, the calculation process of dilated convolution can be represented by the following formula (1): (1) in, The expansion rate; The input feature map; The output feature map is given; the convolution kernel weights are... The kernel size is ; This represents the row index within the convolution kernel window; Indicates the column index within the convolution kernel window; Indicates the row index of the output feature map; Indicates the column index of the output feature map.

[0029] Among them, the dilated convolution calculation method, without changing the number of parameters or reducing the feature map resolution, expands the receptive field from... Expanded to .

[0030] In one feasible implementation, in order to obtain contextual information, it is assumed that the input feature map size is... The number of channels is Then the global average pooling process is represented by the following formula (2): (2) in, Indicates the height of the feature map; Indicates the width of the feature map; Indicates the first c The global maximum response of each channel; This indicates the location of the channel. Activation value; Global average pooling calculates the arithmetic mean of all pixel values ​​in each channel, compressing the H×W spatial features into a 1×1 global vector, thereby giving the network the ability to capture overall environmental information.

[0031] In one feasible implementation, in the steps of the regular convolution branch, the three-branch dilated convolution branch, and the global average pooling branch, the convolution operation is immediately followed by batch normalization and nonlinear activation function processing to prevent gradient vanishing and accelerate convergence. The batch normalization process is represented by the following formulas (3)-(6): (3) (4) (5) (6) in, This represents the mean of a small batch of data; It represents a one-dimensional feature of a small batch of data samples; Indicates the number of samples in a mini-batch; This represents the variance of a small batch of data; To represent a very small constant, preventing division by zero; Indicates the learnable scaling parameter; Represents the learnable translation parameters; This represents the output of the batch normalization layer; This represents the result after normalizing a small batch of data.

[0032] S34. The local feature map, feature maps at different scales and the global feature map are cascaded and stitched together in the channel dimension, and then fused through the projection layer to obtain the SAR image features and the edge contour features between various subjects in the SAR image.

[0033] In one feasible implementation, the process of cascading and splicing the channel dimensions is represented by the following formulas (7)-(8): (7) (8) in, This represents the input feature map; n represents the number of branch feature maps; This represents the feature map after cascading and splicing. () indicates batch normalization; () represents the ReLU activation function; Indicates output layer Convolutional kernels are used to compress the excessive number of channels after concatenation back to the specified number of output channels, enabling cross-branch information exchange. Indicates the bias term; This represents the feature map of the final output.

[0034] The projection layer includes convolution, batch normalization, and nonlinear activation functions.

[0035] In one feasible implementation, the feature extraction encoder includes: a shallow layer (Stage 1), a middle layer (Stage 2), and a deep layer (Stage 3). The output of the shallow layer is upsampled once and convolved once, doubling both its size and number of channels, denoted as e1. The output of the middle layer is upsampled, quadrupling its size to match e1, and convolved to adjust the number of channels to match e1, denoted as e2. e1 and e2 are concatenated to obtain an output containing multi-level information. Because this output is derived from two parts, the number of channels doubles, so it undergoes another convolution to halve the number of channels, denoted as f1. The output of the deep layer is upsampled, quadrupling its size to match e1, and convolved to adjust the number of channels to match e1, denoted as e3. e3 is concatenated with f1. Because this output is derived from two parts, the number of channels doubles, so it undergoes another convolution to halve the number of channels, denoted as f2. f2 is the core output of the edge sensing module. It is used to generate the edge contour prediction map between various subjects in the SAR image and to guide the subsequent decoder.

[0036] In this invention, which addresses the characteristic of feature extraction encoders that progressively abstract semantics from shallow to deep layers, a dynamically increasing dilation rate configuration strategy is designed. The dilation rate combination of the four stages is set to gradually transition from fine-grained detection at small scales to wide-area coverage at large scales. This ensures that the encoder can accurately capture subtle textures at shallow layers and effectively integrate a wide range of spatial semantic information at deep layers, significantly enhancing the model's robustness to multi-scale features.

[0037] The basic features extracted by the initialization layer are input to the decoding layer 1 through skip connections, and the different scale features extracted by the three dilated convolution branches in the encoding layer are input to the decoding layers 2, 3 and 4 through skip connections.

[0038] Optionally, the semantic segmentation loss function of SAR images is the sum of the binary cross-entropy loss function and the DICE loss function of the segmentation results, expressed by the following formulas (9)-(11): (9) (10) (11) in, The total number of pixels contained in this batch of images; This represents the probability value of the foreground target predicted by the semantic segmentation model; Represents the truth value label for semantic segmentation; This represents any single pixel in the input image; To represent a very small constant, preventing division by zero; The binary cross-entropy loss function represents the segmentation result; Represents the DICE loss function; This represents the semantic segmentation loss function for SAR images.

[0039] S4. Input the features of the edge contours into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map, construct the segmentation loss function of the edge contour between each subject in the SAR image.

[0040] In one feasible implementation, the SAR image features E1, E2, and E3 extracted by the three dilated convolution branches in the dilated convolution-based feature extraction encoder are upsampled and convolved at different scales. Then, SAR image features E1 and E2 are concatenated and processed by the convolution module before being concatenated and convolved with SAR image feature E3 to form a feature map with edge information. The feature map with edge information is the core output of the edge perception module, which will be subsequently transmitted back to each decoding layer for fusion with SAR image semantic segmentation information and generation of predicted edge contour maps between various subjects in the SAR image.

[0041] The edge perception module, when processing features, employs a standard combination of operators from deep convolutional neural networks, specifically comprising three parts: (1) The convolution operation is represented by the following formula (12): (12) in, This represents the output feature map; Represents the convolution kernel In input features The gradient information is extracted by performing a sliding window dot product operation. This indicates the bias term.

[0042] (2) Normalization process: The feature distribution is re-standardized by the following formula (13): (13) in, This represents the result after normalization; This represents a one-dimensional input feature of a mini-batch of data; This represents the mean of a small batch of data; This represents the first affine parameter that can be learned; This represents the variance of a small batch of data; To represent a very small constant, preventing division by zero; This represents the first affine parameter that can be learned.

[0043] (3) The modified ReLU activation function of the linear unit is expressed as: It is used to introduce nonlinear factors and eliminate negative responses.

[0044] The output module of the edge sensing module uses the Sigmoid activation function. The logical values ​​of the convolution output are mapped to the (0, 1) interval to generate the final prediction map representing the probability that a pixel is an edge.

[0045] Among them, the edge contour feature map with edge information generated by the edge module is not only used for output edge prediction, but also serves as a global structural guidance signal. It is adaptively downsampled through different numbers of max pooling operations, so as to maintain strict alignment with the feature maps of the decoder in the deep, middle and shallow stages in terms of spatial resolution.

[0046] In one feasible implementation, the edge contour feature f2 output by the edge sensing module is max-pooled and its size is adjusted to match the size of the SAR image feature map (y4) output by the decoder layer 4. The max-pooled output of f2 is denoted as g4. g4 is concatenated with y4 and then compressed through a convolution module to obtain g4'. g4' and y4 are residually concatenated to generate the input y3 for the next stage. The edge sensing module's output feature f2 is max-pooled and its size is adjusted to match the size of the SAR image feature map (y3) output by the decoder layer 3. The output after max pooling is denoted as g3. g3 is concatenated with y3 and then compressed through a convolutional module to obtain g3'. g3' and y3 are then concatenated with a residual to generate the input y2 for the next stage. The output feature f2 of the edge perception module is max pooled and its size is adjusted to match the size of the SAR image feature map (y2) output by the decoder's decoding layer 2. The output of f2 after max pooling is denoted as g2. g2 is concatenated with y2 and then compressed through a convolutional module to obtain g2'. g2' and y2 are then concatenated with a residual and fed into the decoder's decoding layer 1 to generate the final feature y1.

[0047] In one feasible implementation, in order to eliminate the difference in spatial resolution between the output feature maps of different stages of the encoder, the edge perception module uses bilinear interpolation for processing. The specific processing is expressed by the following formula (14): (14) in, Indicates after interpolation The value at; This represents the weighting coefficients calculated based on geometric distance; This represents four pixels in the neighborhood of the original image.

[0048] The segmentation loss of the edge contours between various subjects in the SAR image is a binary cross-entropy loss function.

[0049] Optionally, the segmentation loss function of the edge contours between various subjects in the SAR image is expressed by the following formulas (15)-(16): (15) (16) in, The binary cross-entropy loss function represents the edge contour segmentation result; This indicates the total number of pixels contained in this batch of images; The probability value representing the edge contour; The truth label representing the edge contour; Represents any pixel in the input image

[0050] S5. Input the edge contour features and SAR image features into the decoder for decoding to obtain the SAR image semantic segmentation probability map with fused edge information; construct the SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label.

[0051] Optionally, the specific implementation process of S5 includes S61-S64: S51. Input the edge contour features and SAR image features into the decoder and perform dimension reduction matching through max pooling to obtain the dimension-reduced edge features. In one feasible implementation, the max pooling process is represented by the following formula (17): (17) in, Represents the edge features after dimensionality reduction; The number of pooling iterations is used to ensure that the dimension-reduced edge features are consistent with the feature size of the SAR image. Represents the initial edge features of the input; This represents the max pooling operation; S52. Physically stitch the dimension-reduced edge features with SAR image features along the channel dimension to construct fused features; The physical splicing process is represented by the following formula (18): (18) in, Indicates fusion characteristics; Indicates SAR image features; Indicates the dimension along which the features are concatenated; This indicates a splicing operation.

[0052] S53. Input the fused features into the edge sensing module for nonlinear mapping to obtain the correction signal of the edge information on the SAR image features; In one feasible implementation, the process of obtaining the correction signal is represented by the following formula (19): (19) in, Indicates a correction signal; Indicates the activation function; This indicates a batch normalization operation; Indicates the convolution kernel weight parameters; Represents the feature map; This indicates the bias term.

[0053] In this process, by calculating the correction signal, the semantic segmentation network based on SAR images can determine which semantic segmentation regions need to be enhanced or suppressed based on the edge location.

[0054] S54. Based on the correction signal, it is added back to the SAR image features in the form of residuals to form a SAR image semantic segmentation probability map that fuses edge information.

[0055] In one feasible implementation, the process of adding the corrected signal back to the SAR image features in the form of residuals is represented by the following formula (20): (20) in, This represents the characteristics of the final output; Indicates a correction signal; Indicates SAR image characteristics.

[0056] In one feasible implementation, a semantic segmentation network based on SAR images can perform fine-tuning only in the edge contour regions between various subjects in the SAR image without destroying the original semantic segmentation information.

[0057] S6. Based on the semantic segmentation loss function of SAR images and the segmentation loss function of the edge contours between various subjects in SAR images, construct the total loss function of the semantic segmentation network based on SAR images; train the semantic segmentation network based on SAR images based on the total loss function to obtain the trained semantic segmentation network.

[0058] In one feasible implementation, the semantic segmentation loss function based on SAR images and the segmentation loss function of the edge contours between various subjects in SAR images are combined by a linear weighting method. The total loss function of the semantic segmentation network based on SAR images is expressed by the following formula (21): (twenty one) in, This represents the total loss function of the semantic segmentation network based on SAR images; Represents the weights of the SAR image semantic segmentation loss function; This represents the loss function for semantic segmentation based on SAR images; The segmentation loss weights represent the edge contours between various subjects within a SAR image. This represents the segmentation loss function for the edge contours between various subjects within a SAR image.

[0059] S7. Obtain the SAR image to be detected; input the SAR image to be detected into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

[0060] This invention, by integrating edge perception functionality and introducing a multi-task learning strategy with edge constraints, can fully exploit the edge features of SAR images at different scales and improve the ability to identify complex ground feature boundaries.

[0061] The embodiments of the present invention effectively take into account both local details and global contextual information through multi-scale feature fusion, thereby improving the extraction capability of small targets and large-scale targets.

[0062] The present invention can enhance the binary semantic segmentation and interpretation of various land cover types in SAR images, including but not limited to water body segmentation, building segmentation, road segmentation and other infrastructure segmentation.

[0063] The embodiments of the present invention do not require strict prior assumptions or complex artificial feature design for the input SAR image, and are applicable to SAR data of different resolutions and different imaging modes, with good versatility and scalability.

[0064] Figure 3 This is a block diagram of a semantic segmentation device based on SAR image fusion edge sensing function provided in an embodiment of the present invention. This device is used in a semantic segmentation method based on SAR image fusion edge sensing function. (Refer to...) Figure 3 The device includes an image acquisition unit 310, a first construction unit 320, a feature extraction unit 330, a second construction unit 340, a third construction unit 350, a training unit 360, and a segmentation unit 370. Wherein: Image acquisition unit 310 is used to acquire SAR images and corresponding ground truth labels; and to extract edge contour images between different subjects in the SAR images using the Canny edge detection algorithm. The first construction unit 320 is used to construct a semantic segmentation network based on SAR images; wherein, the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge awareness module, and a decoder with feature fusion function; Feature extraction unit 330 is used to input SAR image and edge contour image into feature extraction encoder for feature extraction, and obtain SAR image features and edge contour features between various subjects in SAR image; The second construction unit 340 is used to input the features of the edge contour into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; and to construct the segmentation loss function of the edge contour between each subject in the SAR image based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map. The third construction unit 350 is used to input the features of the edge contour and the SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information; and to construct a SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label. Training unit 360 is used to construct the total loss function of the semantic segmentation network based on SAR image based on the semantic segmentation loss function of SAR image and the segmentation loss function of the edge contours between various subjects in SAR image; and to train the semantic segmentation network based on SAR image based on the total loss function to obtain the trained semantic segmentation network. The segmentation unit 370 is used to acquire the SAR image to be detected; the SAR image to be detected is input into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

[0065] Optionally, the CNN-based feature extraction encoder that incorporates dilated convolutions adopts a fully embedded dilated spatial pyramid feature extraction architecture to replace the traditional single convolution stacking mode. The feature extraction architecture includes four consecutive downsampling stages; each stage integrates a hollow spatial pyramid pooling module; and each module constructs five parallel feature extraction branches. Among them, the five parallel feature extraction branches include: one regular convolution branch, three dilated convolution branches with different dilation rate parameters, and one image-level context-aware branch containing global average pooling operation. Among them, the conventional convolutional branch is used to preserve the inherent local details of the features; Among them, the dilated convolution branch is used to expand the receptive field through sparse sampling without reducing the resolution; Among them, the edge perception module is used to extract the edge contour features of each subject in the SAR image.

[0066] Optionally, the feature extraction unit 330 is used for: The SAR image and the edge contour image are input into the initialization layer to extract features, and the basic feature map is output. The basic feature map is input into a regular convolutional branch, and the local feature map is output. The basic feature map is input into three dilated convolution branches to extract contextual information at different scales, and feature maps at different scales are output. The basic feature map is input into the image-level context-aware branch containing global average pooling operation to extract global semantic information and output a global feature map. Local feature maps, feature maps at different scales, and global feature maps are cascaded and stitched together along the channel dimension, and then fused through a projection layer to obtain SAR image features and the edge contour features between various subjects within the SAR image.

[0067] Optionally, the calculation process of the dilated convolution is represented by the following formula (1): (1) in, The expansion rate; The input feature map; The output feature map is given; the convolution kernel weights are... The kernel size is ; This represents the row index within the convolution kernel window; Indicates the column index within the convolution kernel window; Indicates the row index of the output feature map; Indicates the column index of the output feature map.

[0068] Optionally, the third building unit 350 is used for: The edge contour features and SAR image features are input into the decoder and dimensionality reduction matching is performed by max pooling to obtain the dimensionality-reduced edge features. The reduced edge features are physically stitched together with SAR image features along the channel dimension to construct fused features; The fused features are input into the edge sensing module for nonlinear mapping to obtain the correction signal of the edge information to the SAR image features; The corrected signal is added back to the SAR image features in the form of residuals to form a SAR image semantic segmentation probability map that fuses edge information.

[0069] Optionally, the SAR image semantic segmentation loss function Binary cross-entropy loss function and DICE loss function of the segmentation results This can be expressed by the following formulas (2)-(4): (2) (3) (4) in, This indicates the total number of pixels contained in this batch of images; This represents the probability value of the foreground target predicted by the semantic segmentation model; Represents the truth value label for semantic segmentation; This represents any single pixel in the input image; To represent a very small constant, preventing division by zero; This represents the semantic segmentation loss function for SAR images; The binary cross-entropy loss function represents the segmentation result; This represents the DICE loss function.

[0070] Optionally, the segmentation loss function for the edge contours between subjects in the SAR image is expressed by the following formula (5): (5) in, The segmentation loss function represents the edge contours between various subjects within a SAR image; The binary cross-entropy loss function represents the edge contour segmentation result; This indicates the total number of pixels contained in this batch of images; The probability value representing the edge contour; The truth label representing the edge contour; This represents any single pixel in the input image.

[0071] This invention, by integrating edge perception functionality and introducing a multi-task learning strategy with edge constraints, can fully exploit the edge features of SAR images at different scales and improve the ability to identify complex ground feature boundaries.

[0072] The embodiments of the present invention effectively take into account both local details and global contextual information through multi-scale feature fusion, thereby improving the extraction capability of small targets and large-scale targets.

[0073] The present invention can enhance the binary semantic segmentation and interpretation of various land cover types in SAR images, including but not limited to water body segmentation, building segmentation, road segmentation and other infrastructure segmentation.

[0074] The embodiments of the present invention do not require strict prior assumptions or complex artificial feature design for the input SAR image, and are applicable to SAR data of different resolutions and different imaging modes, with good versatility and scalability.

[0075] Figure 4 This is a schematic diagram of the structure of a semantic segmentation device based on SAR image fusion edge perception function provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a semantic segmentation device based on SAR image fusion edge perception can include the above-mentioned... Figure 3 The semantic segmentation device based on SAR image fusion edge perception function is shown. Optionally, the semantic segmentation device 410 based on SAR image fusion edge perception function may include a first processor 2001.

[0076] Optionally, the semantic segmentation device 410 based on SAR image fusion edge perception function may also include a memory 2002 and a transceiver 2003.

[0077] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0078] The following is combined Figure 4 The following is a detailed introduction to the various components of the semantic segmentation device 410 based on SAR image fusion edge perception: The first processor 2001 is the control center of the semantic segmentation device 410 based on SAR image fusion edge perception function. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0079] Optionally, the first processor 2001 can execute various functions of the semantic segmentation device 410 based on SAR image fusion edge perception by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0080] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0081] In a specific implementation, as one example, the semantic segmentation device 410 based on SAR image fusion edge perception function may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0082] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0083] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the semantic segmentation device 410 based on SAR image fusion edge perception function. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0084] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0085] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0086] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the semantic segmentation device 410 based on SAR image fusion edge perception function. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0087] It should be noted that, Figure 4 The structure of the semantic segmentation device 410 based on SAR image fusion edge awareness shown in the figure does not constitute a limitation on the router. Actual semantic segmentation devices based on SAR image fusion edge awareness may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0088] Furthermore, the technical effect of the semantic segmentation device 410 based on the edge perception function of SAR image fusion can be referred to the technical effect of the semantic segmentation method based on the edge perception function of SAR image fusion described in the above method embodiments, and will not be repeated here.

[0089] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0090] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0091] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0092] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0093] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0094] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0100] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A semantic segmentation method based on SAR image fusion edge sensing function, characterized in that, The method includes: S1. Obtain SAR images and corresponding ground truth labels; use the Canny edge detection algorithm to extract edge contour images between different subjects in the SAR images; S2. Construct a semantic segmentation network based on SAR images; wherein, the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge-aware module, and a decoder with feature fusion function; S3. Input the SAR image and the edge contour image into the feature extraction encoder to extract features, and obtain SAR image features and edge contour features between various subjects in the SAR image. S4. Input the features of the edge contour into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map, construct the segmentation loss function of the edge contour between each subject in the SAR image. S5. Input the features of the edge contour and the SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information; construct a SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label; Specifically, step S5 involves inputting the features of the edge contour and SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information, including: S51. Input the edge contour features and SAR image features into the decoder and perform dimension reduction matching by max pooling to obtain the dimension-reduced edge features; S52. Physically stitch the dimension-reduced edge features with SAR image features along the channel dimension to construct fused features; S53. Input the fused features into the edge sensing module for nonlinear mapping to obtain the correction signal of the edge information on the SAR image features; S54. Based on the correction signal, it is added back to the SAR image features in the form of residuals to form a SAR image semantic segmentation probability map that fuses edge information. S6. Based on the semantic segmentation loss function of SAR image and the segmentation loss function of the edge contours between various subjects in the SAR image, construct the total loss function of the semantic segmentation network based on SAR image; train the semantic segmentation network based on SAR image based on the total loss function to obtain the trained semantic segmentation network. S7. Obtain the SAR image to be detected; input the SAR image to be detected into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

2. The semantic segmentation method based on SAR image fusion edge perception function according to claim 1, characterized in that, The CNN-based feature extraction encoder that incorporates dilated convolutions adopts a fully embedded dilated spatial pyramid feature extraction architecture to replace the traditional single convolution stacking mode. The feature extraction architecture includes four consecutive downsampling stages; each stage integrates a hollow spatial pyramid pooling module; and each module constructs five parallel feature extraction branches. Among them, the five parallel feature extraction branches include: one regular convolution branch, three dilated convolution branches with different dilation rate parameters, and one image-level context-aware branch containing global average pooling operation. Among them, the conventional convolutional branch is used to preserve the inherent local details of the features; Among them, the dilated convolution branch is used to expand the receptive field through sparse sampling without reducing the resolution; Among them, the edge perception module is used to extract the edge contour features of each subject in the SAR image.

3. The semantic segmentation method based on SAR image fusion edge perception function according to claim 2, characterized in that, S3 involves inputting the SAR image and the edge contour image into the feature extraction encoder for feature extraction, obtaining SAR image features and the edge contour features between various subjects within the SAR image, including: S31. Input the SAR image and the edge contour image into the initialization layer to extract features and output the basic feature map; S32. Input the basic feature map into the regular convolutional branch and output the local feature map; S33. Input the basic feature map into the three dilated convolution branches respectively to extract contextual information at different scales and output feature maps at different scales. S34. Input the basic feature map into the image-level context-aware branch containing the global average pooling operation to extract global semantic information and output the global feature map. S35. The local feature map, feature maps at different scales and the global feature map are cascaded and stitched together in the channel dimension, and then fused through the projection layer to obtain the SAR image features and the edge contour features between various subjects in the SAR image.

4. The semantic segmentation method based on SAR image fusion edge perception function according to claim 3, characterized in that, The calculation process of the dilated convolution is represented by the following formula (1): （1） in, The expansion rate; The input feature map; The output feature map is given; the convolution kernel weights are... The kernel size is ; Indicates the row index within the convolution kernel window; Indicates the column index within the convolution kernel window; Indicates the row index of the output feature map; Indicates the column index of the output feature map.

5. The semantic segmentation method based on SAR image fusion edge perception function according to claim 1, characterized in that, The SAR image semantic segmentation loss function is the sum of the binary cross-entropy loss function and the DICE loss function of the segmentation result, expressed by the following formulas (2)-(4): （2） （3） （4） in, This indicates the total number of pixels contained in this batch of images; This represents the probability value of the foreground target predicted by the semantic segmentation model; Represents the truth value label for semantic segmentation; This represents any single pixel in the input image; To represent a very small constant, preventing division by zero; This represents the semantic segmentation loss function for SAR images; The binary cross-entropy loss function represents the segmentation result; This represents the DICE loss function.

6. The semantic segmentation method based on SAR image fusion edge perception function according to claim 1, characterized in that, The segmentation loss function of the edge contours between various subjects in the SAR image is expressed by the following formula (5): （5） in, The segmentation loss function represents the edge contours between various subjects within a SAR image; The binary cross-entropy loss function represents the edge contour segmentation result; This indicates the total number of pixels contained in this batch of images; The probability value representing the edge contour; The truth label representing the edge contour; This represents any single pixel in the input image.

7. A semantic segmentation device based on SAR image fusion edge perception function, wherein the semantic segmentation device based on SAR image fusion edge perception function is used to implement the semantic segmentation method based on SAR image fusion edge perception function as described in any one of claims 1-6, characterized in that, The device includes: The image acquisition unit is used to acquire SAR images and corresponding ground truth labels; the Canny edge detection algorithm is used to extract the edge contour images between different subjects in the SAR images; The first building unit is used to build a semantic segmentation network based on SAR images; wherein, the semantic segmentation network includes: an initialization layer, a feature extraction encoder based on CNN and incorporating dilated convolution, an edge awareness module, and a decoder with feature fusion function; The feature extraction unit is used to input the SAR image and the edge contour image into the feature extraction encoder for feature extraction, and to obtain SAR image features and edge contour features between various subjects in the SAR image. The second construction unit is used to input the features of the edge contour into the edge perception module for processing to obtain the edge contour segmentation probability map between each subject in the SAR image; and to construct the segmentation loss function of the edge contour between each subject in the SAR image based on the edge contour segmentation probability map and the ground truth label corresponding to the edge contour map. The third construction unit is used to input the features of the edge contour and the SAR image features into the decoder for decoding to obtain a SAR image semantic segmentation probability map with fused edge information; and to construct a SAR image semantic segmentation loss function based on the semantic segmentation probability map and its corresponding ground truth label. The training unit is used to construct the total loss function of the semantic segmentation network based on SAR images based on the semantic segmentation loss function of SAR images and the segmentation loss function of the edge contours between various subjects in SAR images; and to train the semantic segmentation network based on SAR images based on the total loss function to obtain the trained semantic segmentation network. The segmentation unit is used to acquire the SAR image to be detected; the SAR image to be detected is input into the trained semantic segmentation network to obtain the semantic segmentation result of the SAR image.

8. A semantic segmentation device based on SAR image fusion edge perception function, characterized in that, The semantic segmentation device based on SAR image fusion edge perception function includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images