Border-optimized remote sensing image semantic segmentation method and device, equipment and medium

Abstract

This invention proposes a boundary-optimized semantic segmentation method, apparatus, device, and medium for remote sensing images. By constructing a semantic segmentation network for remote sensing images comprising an encoder, a boundary detection branch network, and a boundary point upsampling module, in the encoding stage, the boundary detection branch network uses gated convolution to control feature map information, making the segmentation network pay more attention to pixels in boundary regions during the learning process, thus enhancing the encoding of boundary information. In the decoding stage, a boundary point upsampling module is designed for pixels in boundary regions of the remote sensing image. A separate small network is used to train the sampled pixels, and the segmentation results of the sampled pixels are predicted by the small network. The segmentation results of other regions are obtained by direct interpolation of deep feature maps. This invention achieves more refined semantic segmentation results while reducing model parameters and computational load.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, apparatus, device, and medium for semantic segmentation of remote sensing images with boundary optimization. Background Technology

[0002] The purpose of image segmentation is to label each pixel as a category. For remote sensing images, this means labeling pixels as a type of land cover, such as buildings, water bodies, roads, farmland, vehicles, etc. Image semantic segmentation evolved from traditional image segmentation methods. Traditional methods (thresholding, k-Means clustering, region mapping, edge detection) only focus on finding the boundaries of land covers, not their categories. Semantic segmentation, however, not only accurately finds the outlines of land covers but also accurately determines their categories, i.e., provides their semantic meaning. Thanks to the rapid development of deep learning and the significant improvement in computer storage and computing power, semantic segmentation methods based on deep convolutional neural networks have become a powerful new tool for high-resolution remote sensing image segmentation.

[0003] Fully convolutional networks (FCNs) were the first to achieve an end-to-end semantic segmentation method based on deep convolutional neural networks. However, the segmentation results of fully convolutional networks have relatively coarse boundaries, especially the inability to fully recover object boundary information. The main reasons for this are: image semantic segmentation is a pixel-level classification task, and label creation is difficult and costly. Remote sensing images contain rich ground feature information. Since the deep features obtained by convolutional neural networks through multiple convolution and pooling calculations are abstract, they are obtained by direct upsampling during the process of restoring to the original image size, or only aggregating a portion of the low-level feature maps before upsampling to the original image size. This fails to fully integrate the low-level fine-grained feature maps, resulting in a lack of detailed information and inaccurate prediction results for the boundary parts.

[0004] To address the aforementioned shortcomings, this paper utilizes the concept of UNet (a classic semantic segmentation network) to design a symmetrical encoder and decoder structure. The encoder performs multiple convolutions and pooling, while the decoder employs multiple skip connections for upsampling. Each upsampling process fuses feature maps of the same size from the encoder until the encoder's output is restored to the original image size. This fully utilizes feature maps at various scales, enabling the recovery of detailed information about ground feature boundaries in the semantic segmentation result. However, the UNet decoder generates redundant feature maps after multiple upsamplings, resulting in more network parameters and inevitably increasing the network's computational cost. Furthermore, it does not perform any special optimization for semantic segmentation boundaries. Summary of the Invention

[0005] This invention provides a boundary-optimized semantic segmentation method for remote sensing images, which aims to reduce network model parameters and computational load, and obtain more refined semantic segmentation results.

[0006] Therefore, the purpose of this invention is to propose a boundary-optimized semantic segmentation method for remote sensing images, comprising:

[0007] A remote sensing semantic segmentation network is constructed, which is an improvement based on the classic semantic segmentation network U-NET.

[0008] A remote sensing semantic segmentation network was trained using a remote sensing image dataset;

[0009] The real-time captured remote sensing images are input into the trained remote sensing image semantic segmentation network, and the output result is the semantic segmentation result of the captured remote sensing images.

[0010] The steps for training the remote sensing semantic segmentation network using a remote sensing image dataset include:

[0011] Obtain a remote sensing image sample dataset, label the land cover categories of the remote sensing images in the dataset by pixels, generate corresponding labeled remote sensing images, and divide them into training set, validation set and test set after preprocessing;

[0012] Construct a semantic segmentation network for remote sensing images, which includes an encoder, a boundary detection branch network, and a boundary point upsampling module;

[0013] The original remote sensing images of the training set are input into the remote sensing image semantic segmentation network for training. The hyperparameters and loss function of the model training are determined, and the parameters of the remote sensing image semantic segmentation network are optimized until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold.

[0014] The test set is input into the trained remote sensing image semantic segmentation network to verify the network's segmentation accuracy.

[0015] The training method of the remote sensing semantic segmentation network is as follows: the training set of remote sensing images is input into the encoder, the feature maps output by different convolutional layers are input into the boundary detection branch network, the output results of the encoder and the boundary detection branch network are fused to obtain the preliminary segmented remote sensing image, the boundary point upsampling module samples pixels based on the preliminary segmented remote sensing image, and the sampling points, the feature maps output by each convolutional layer of the encoder, and the original remote sensing image input are combined by a small network formed by a multilayer perceptron to make predictions. The output results are matched and replaced with the preliminary segmented remote sensing image to obtain the boundary-optimized remote sensing image semantic segmentation result.

[0016] The preprocessing steps for remote sensing images and their corresponding labeled remote sensing images include:

[0017] Random sampling is performed on the original remote sensing image and the corresponding labeled remote sensing image. Each sampling yields a 512×512 remote sensing image and the corresponding labeled image.

[0018] Sample enhancement is performed on the remote sensing images obtained after random sampling, including random rotation of n×90° (n=0,1,2,3), random 90° flipping in the horizontal and vertical directions, random scaling, and random enhancement of brightness, contrast, and saturation.

[0019] The encoder uses ResNet as its backbone network, which includes several sequentially connected convolutional layers. The feature map output from the previous convolutional layer is input into the next convolutional layer for feature extraction to generate a new feature map.

[0020] The boundary detection branch network includes the same number of feature extraction layers as the encoder convolutional layers. Each feature extraction layer includes a gated convolution and an upsampling mechanism to control the flow of information. The input of the gated convolution of each feature extraction layer is connected to one of the convolutional layers of the encoder, and the feature map output by the corresponding convolutional layer is used as the gated convolution input of the corresponding feature extraction layer. After the feature map of each convolutional layer is upsampled by the corresponding gated convolution, feature maps of different scales are formed and fused with the feature map output by the encoder. The result is a preliminary segmentation of the remote sensing image.

[0021] The gated convolution process is shown in formula (1):

[0022]

[0023] Where: X is the input feature map, i and j represent the position of each pixel, and σ is the sigmoid function.

[0024] The boundary point sampling module samples the initially segmented remote sensing image and collects pixels with confidence scores below a preset threshold as sampling points. The sampling points are matched and fused with feature maps of different scales output by each convolutional layer of the encoder and the original remote sensing image. The fusion result is input into a small network, and the output result is the prediction result of the sampling points. The prediction result of the sampling points is matched and replaced with the initially segmented remote sensing image to obtain the boundary-optimized semantic segmentation result of the remote sensing image.

[0025] The sampling process of the boundary point sampling module for sampling the initially segmented remote sensing image is as follows:

[0026] Based on the preliminary segmentation of the remote sensing image, the predicted probability value of each pixel in the image is calculated as shown in formula (2):

[0027] P=θ(X) (2)

[0028] image Inputting the remote sensing image into the semantic segmentation network θ yields the initial predicted probability map. K is the number of categories. This represents the pixel in the i-th row and j-th column of X. Let P be the predicted probability of the pixel in the i-th row and j-th column; sort all pixels in the predicted probability map P in descending order of probability to obtain the sorted probability map.

[0029] The negative of the difference between the highest probability P_sorted[1] and the second highest probability P_sorted[2] of all pixel classifications is obtained by formula (3). This is called uncertainty, as shown in formula (4);

[0030] P_sorted = sort(P) (3)

[0031] U=-(P_sorted[1]-P_sorted[2]) (4)

[0032] u i，j U represents the uncertainty of the i-th row and j-th pixel. i，j The larger the value, the higher the uncertainty of the prediction result for that pixel.

[0033] Select the β×N most uncertain pixels, where β∈[0,1], and then uniformly sample the remaining (1-β)×N pixels in the image.

[0034] The steps for predicting sampling points include:

[0035] Based on the initial segmented remote sensing image, feature reconstruction is performed on the sampled uncertain pixels by combining fine-grained features, as expressed in formula (5):

[0036] G = L 1 ||L 2 ||L 3 ||X (5)

[0037] Among them, fine-grained features Low-level feature maps from the encoder and input The encoder is configured with three convolutional layers, and feature maps at three scales are interpolated onto the original remote sensing image to obtain the following results sequentially. M = C1 + C2 + C3 + C; The feature maps are fused with the input X to obtain the reconstructed fine-grained feature representation, where || represents the feature map concatenation operation;

[0038] By concatenating the fine-grained feature G with the predicted probability map P using formula (6), the feature representation of the sampled points reconstructed is obtained. Extract the reconstructed feature representation of each sampling point based on the coordinate information of the sampled pixels;

[0039] R = G||P (6)

[0040] The reconstructed feature vector is used as input to a small network. The predicted results of the output sampling points are matched and replaced with the initially segmented remote sensing image to obtain the semantic segmentation result of the remote sensing image with optimized boundaries.

[0041] The hyperparameters for model training include the Adam optimizer, an exponential decay rate of (0.9, 0.99), a regularization coefficient of 0.0002, and a learning rate of 0.001. Since it involves two tasks—semantic segmentation and boundary detection—and also requires training a small network in the boundary point upsampling module, the overall loss function consists of three parts, as shown in formula (7). The semantic segmentation task and the small network are trained using the cross-entropy loss function, while the boundary detection branch network is trained using the binary cross-entropy loss function.

[0042]

[0043] θ and δ represents the parameters of the semantic segmentation backbone network and the boundary detection branch network, respectively, and δ represents the parameters of the smaller network. 'b' represents the semantic boundary label, and 'b' represents the prediction result of the boundary detection branch network. y represents the semantic segmentation label, and y represents the prediction result of the semantic segmentation network. Let z be the label of the uncertain points sampled by the boundary point sampling module, z be the prediction result of the small network, and λ1 and λ2 be hyperparameters that adjust the weight of semantic segmentation loss and boundary extraction loss, respectively, which are 10 and 1.

[0044] Furthermore, the present invention also aims to provide a boundary-optimized remote sensing image semantic segmentation apparatus, which uses the boundary-optimized remote sensing image semantic segmentation method as described above for segmentation, including:

[0045] The network construction module is used to build a remote sensing semantic segmentation network, which is an improvement on the classic semantic segmentation network U-NET.

[0046] The network training module is used to train the remote sensing semantic segmentation network using a remote sensing image dataset;

[0047] The image segmentation module is used to input real-time captured remote sensing images into the trained remote sensing image semantic segmentation network, and the output result is the semantic segmentation result of the captured remote sensing images.

[0048] Another objective of this invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the methods described in the foregoing technical solutions.

[0049] Another objective of this invention is to provide a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the method of the aforementioned technical solution.

[0050] Unlike existing technologies, the boundary-optimized semantic segmentation method for remote sensing images provided in this invention constructs a semantic segmentation network for remote sensing images, including an encoder, a boundary detection branch network, and a boundary point upsampling module. In the encoding stage, the boundary detection branch network uses gated convolution to control feature map information, making the segmentation network pay more attention to pixels in boundary regions during the learning process, thus enhancing the encoding of boundary information. In the decoding stage, a boundary point upsampling module is designed for pixels in boundary regions of the remote sensing image. A separate small network is used to train the sampled pixels, and the segmentation results of the sampled pixels are predicted by the small network. The segmentation results of other regions are obtained by direct interpolation of deep feature maps. This invention achieves more refined semantic segmentation results while reducing model parameters and computational load. Attached Figure Description

[0051] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0052] Figure 1 This is a flowchart illustrating a boundary-optimized semantic segmentation method for remote sensing images provided by the present invention.

[0053] Figure 2 This is a schematic diagram of the network structure of a boundary-optimized semantic segmentation method for remote sensing images provided by the present invention.

[0054] Figure 3 This is a flowchart illustrating the upsampling operation in a boundary-optimized semantic segmentation method for remote sensing images provided by the present invention.

[0055] Figure 4 This is a flowchart illustrating the gated convolution operation in a boundary-optimized remote sensing image semantic segmentation method provided by the present invention. Detailed Implementation

[0056] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0057] The following describes a boundary-optimized semantic segmentation method for remote sensing images according to an embodiment of the present invention, with reference to the accompanying drawings.

[0058] Figure 1This is a flowchart illustrating a boundary-optimized semantic segmentation method for remote sensing images provided in an embodiment of the present invention. The method includes the following steps:

[0059] Step 101: Construct a remote sensing semantic segmentation network, which is an improvement on the classic semantic segmentation network U-NET.

[0060] A remote sensing semantic segmentation network is constructed, comprising an encoder, a boundary detection branch network, and a boundary point upsampling module. The network structure is as follows: Figure 2 As shown.

[0061] The encoder uses ResNet as its backbone network, comprising several sequentially connected convolutional layers. The feature map output from the previous convolutional layer is input into the next convolutional layer for feature extraction, generating a new feature map. In this embodiment of the invention, the number of convolutional layers is set to 3.

[0062] The boundary detection branch network includes the same number of feature extraction layers as the encoder convolutional layers. Each feature extraction layer includes gated convolution and upsampling mechanisms to control the flow of information. The input of the gated convolution of each feature extraction layer is connected to one of the encoder's convolutional layers, and the feature map output of the corresponding convolutional layer is used as the gated convolution input of the corresponding feature extraction layer. After the feature map of each convolutional layer is interpolated and upsampled by the corresponding gated convolution, feature maps of different scales are formed and fused with the feature map output by the encoder. The result is a preliminary segmentation of the remote sensing image. The process of gated convolution and upsampling operations is as follows: Figure 3 and Figure 4 As shown.

[0063] The process of gated convolution is shown in equation (1):

[0064]

[0065] Where: X is the input feature map, i and j represent the position of each pixel, and σ is the sigmoid function.

[0066] The boundary point sampling module samples the pre-segmented remote sensing image and collects pixels with confidence scores below a preset threshold as sampling points. The sampling points are matched and fused with feature maps of different scales output by each convolutional layer of the encoder and the original remote sensing image. The fusion result is input into a small network, and the output result is the prediction result of the sampling points. The prediction result of the sampling points is matched and replaced with the pre-segmented remote sensing image to obtain the semantically segmented remote sensing image.

[0067] Step 102: Train the remote sensing semantic segmentation network using the remote sensing image dataset.

[0068] The steps for training the remote sensing semantic segmentation network using a remote sensing image dataset include:

[0069] Obtain a remote sensing image sample dataset, label the land cover categories of the remote sensing images in the dataset by pixels, generate corresponding labeled remote sensing images, and automatically generate semantic boundary labels by using an edge detection algorithm on the labeled remote sensing images; after preprocessing, divide the dataset into training set, validation set and test set;

[0070] Construct a semantic segmentation network for remote sensing images, which includes an encoder, a boundary detection branch network, and a boundary point upsampling module;

[0071] The original remote sensing images of the training set are input into the remote sensing image semantic segmentation network for training. The hyperparameters and loss function of the model training are determined, and the parameters of the remote sensing image semantic segmentation network are optimized until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold.

[0072] The test set is input into the trained remote sensing image semantic segmentation network to verify the network's segmentation accuracy.

[0073] A certain number of remote sensing images are extracted from the remote sensing image database to construct a sample dataset of remote sensing images. The land cover categories in the remote sensing images are manually labeled by pixel to obtain the corresponding labeled images. In other embodiments, labeled remote sensing images with the boundary contours of land covers can also be directly obtained from a network image database. The labeled remote sensing images are added to the remote sensing image dataset, and data augmentation operations are performed on the obtained remote sensing image dataset.

[0074] Data augmentation begins by increasing the number of samples. Specifically, this is achieved by randomly sampling the original remote sensing images and their corresponding labeled remote sensing images from the remote sensing image dataset, resulting in smaller images smaller than the original remote sensing images. In this embodiment, through random sampling, the original remote sensing images and their corresponding labeled remote sensing images are cropped to a size of 512×512, smaller than the original size, thus achieving data augmentation.

[0075] After random sampling, the parameters of the entire remote sensing image sample dataset are further adjusted, specifically including: random n×90° rotation (n=0, 1, 2, 3), random 90° horizontal and vertical flips, random scaling from 0.5x to 1.5x, random brightness enhancement from 0.5x to 1.5x, random contrast enhancement from 0.5x to 1.5x, and random saturation enhancement from 0.5x to 1.5x. Each sample undergoes random number and degree of data augmentation to enrich the training sample set to the greatest extent possible and improve the generalization ability of the deep semantic segmentation network. After each iteration, the accuracy of the deep semantic segmentation network is verified using a validation dataset to obtain the deep semantic segmentation network with the highest accuracy. Inputting a high-resolution remote sensing image from the test set into the obtained deep semantic segmentation network yields the semantic segmentation result for that image.

[0076] In addition, preprocessing methods also include conventional operations such as deduplication, filtering, noise reduction, and grayscale conversion.

[0077] After preprocessing, the preprocessed remote sensing image dataset is divided into training set, validation set and test set according to a preset ratio.

[0078] The hyperparameters for model training include using the Adam optimizer, exponential decay rate, regularization coefficient, or learning rate. Since it involves two tasks, semantic segmentation and boundary detection, and also requires training a small network in the boundary point upsampling module, the overall loss function consists of three parts, as shown in Equation (7). The semantic segmentation task and the small network are trained using the cross-entropy loss function, while the boundary detection branch network is trained using the binary cross-entropy loss function.

[0079]

[0080] θ and δ represents the parameters of the semantic segmentation backbone network and the boundary detection branch network, respectively, and δ represents the parameters of the smaller network. 'b' represents the semantic boundary label, and 'b' represents the prediction result of the boundary detection branch network. y represents the semantic segmentation label, and y represents the prediction result of the semantic segmentation network. λ1 and λ2 are the labels of the uncertain points sampled by the boundary point sampling module, z is the prediction result of the small network, and λ1 and λ2 are hyperparameters that adjust the weight of semantic segmentation loss and boundary extraction loss.

[0081] The training method for the remote sensing image semantic segmentation network is as follows: the training set of remote sensing images is input into the encoder, the feature maps output from different convolutional layers are input into the boundary detection branch network, the output results of the encoder and the boundary detection branch network are fused to obtain a preliminary segmented remote sensing image, the boundary point upsampling module samples pixels based on the preliminary segmented remote sensing image, and the sampling points, the feature maps output from each convolutional layer of the encoder, and the original remote sensing image input are combined by a small network formed by a multilayer perceptron to make predictions, and the output results are matched and replaced with the preliminary segmented remote sensing image to obtain the boundary-optimized remote sensing image semantic segmentation result.

[0082] Specifically, the encoder has three convolutional layers, and the boundary detection branch network has three corresponding feature extraction layers. The remote sensing image data from the training set is input into the encoder, and features are extracted sequentially through each convolutional layer. After processing by the three convolutional layers, corresponding feature maps are obtained. The feature maps extracted by each convolutional layer serve as input to the corresponding feature extraction layer of the boundary detection branch network. After gated convolution processing by the corresponding feature extraction layer, an upsampling operation is performed. The feature maps output from the three feature extraction layers are fused, and then fused with the feature map obtained from the third convolutional layer of the encoder after the upsampling operation. After fusion, an upsampling operation is performed again to generate a preliminary segmented remote sensing image. Finally, the feature maps output from the three feature extraction layers, after fusion and upsampling, generate the boundary detection image of the corresponding remote sensing image.

[0083] Sampling is performed on the preliminarily segmented remote sensing image to obtain sampling points.

[0084] The sampling process of the boundary point sampling module for sampling the initially segmented remote sensing image is as follows:

[0085] Based on the preliminary segmentation of the remote sensing image, the predicted probability value of each pixel in the image is calculated as shown in formula (2):

[0086] P=θ(X) (2)

[0087] image Inputting the remote sensing image into the semantic segmentation network θ yields the initial predicted probability map. K is the number of categories. This represents the pixel in the i-th row and j-th column of X. Let P be the predicted probability of the pixel in the i-th row and j-th column; sort all pixels in the predicted probability map P in descending order of probability to obtain the sorted probability map.

[0088] The negative of the difference between the highest probability P_sorted[1] and the second highest probability P_sorted[2] of all pixel classifications is obtained by formula (3). This is called uncertainty, as shown in formula (4);

[0089] P_sorted = sort(P) (3)

[0090] U=(P_sorted[1]-P_sorted[2]) (4)

[0091] u i，j U represents the uncertainty of the i-th row and j-th pixel. i，j The larger the value, the higher the uncertainty of the prediction result for that pixel.

[0092] Select the β×N most uncertain pixels, where β∈[0,1], and then uniformly sample the remaining (1-β)×N pixels in the image. The larger β is, the more the sampling points are concentrated in the boundary region of the image.

[0093] After the sampling points are acquired, they are fused and matched with the original remote sensing image and the two-dimensional feature map output by the three convolutional layers of the encoder. The features of the sampling points are jointly represented by the feature map in the encoder, the input image and the initial segmentation result. The features of the sampling points are input into a small network, and the output result is the semantic segmentation result of the remote sensing image.

[0094] The steps for sampling point prediction include:

[0095] Since the initial segmentation result did not fuse the low-level feature map, but was obtained by interpolation of the high-level feature map, the segmentation result of the high-frequency region was very coarse. In order to recover the detailed information of the semantic segmentation result, the boundary point upsampling module reconstructed the features of the sampled uncertain pixels based on the initial segmented remote sensing image and combined with fine-grained features. The formula is expressed as formula (5):

[0096] G = L 1 ||L 2 ||L 3 ||X (5)

[0097] Among them, fine-grained features Low-level feature maps from the encoder and input The encoder is configured with three convolutional layers, and feature maps at three scales are interpolated onto the original remote sensing image to obtain the following results sequentially. M = C1 + C2 + C3 + C; The feature maps are fused with the input X to obtain the reconstructed fine-grained feature representation, where || represents the feature map concatenation operation;

[0098] By concatenating the fine-grained feature G with the predicted probability map P using formula (6), the feature representation of the sampled points reconstructed is obtained. Extract the reconstructed feature representation of each sampling point based on the coordinate information of the sampled pixels;

[0099] R = G||P (6)

[0100] The reconstructed feature vector is used as the input of a small network. The predicted results of the output sampling points are matched and replaced with the initially segmented remote sensing image to obtain the semantic segmentation result of the remote sensing image.

[0101] After the network training is completed, the test set is input into the trained remote sensing image semantic segmentation network to verify the network's segmentation accuracy.

[0102] The loss function for the boundary detection branch network, the boundary point upsampling module, and the small network is the cross-entropy loss function. The number of categories in the network output layer is set to the number of categories of the included land features based on the number of categories required to be classified in the dataset. The learning rate is set to 0.0001, the number of iterations is 1500, the exponential decay rate is (0.9, 0.99), and the regularization coefficient is 0.0002.

[0103] The network structure of a small network is shown in Table 1:

[0104]

[0105] Table 1 Small Network Structure

[0106] Step 103: Input the real-time captured remote sensing image into the trained remote sensing image semantic segmentation network, and the output result is the semantic segmentation result of the captured remote sensing image.

[0107] Furthermore, the present invention also aims to provide a boundary-optimized remote sensing image semantic segmentation apparatus, which uses the boundary-optimized remote sensing image semantic segmentation method as described above for segmentation, including:

[0108] The network construction module is used to build a remote sensing semantic segmentation network, which is an improvement on the classic semantic segmentation network U-NET.

[0109] The network training module is used to train the remote sensing semantic segmentation network using a remote sensing image dataset;

[0110] The image segmentation module is used to input real-time captured remote sensing images into the trained remote sensing image semantic segmentation network, and the output result is the semantic segmentation result of the captured remote sensing images.

[0111] The implementation process of the above-mentioned device is similar to or even the same as the implementation process of the semantic segmentation method in the foregoing embodiments, and will not be repeated here.

[0112] The present invention also aims to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a remote sensing image semantic segmentation method with boundary optimization as described in any of the foregoing technical solutions.

[0113] Another objective of this invention is to propose a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a remote sensing image semantic segmentation method with boundary optimization as described above.

[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0116] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0118] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0120] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A boundary-optimized semantic segmentation method for remote sensing images, characterized in that, include: A remote sensing semantic segmentation network is constructed, which is an improvement based on the classic semantic segmentation network U-NET. Training the remote sensing semantic segmentation network using a remote sensing image dataset includes: acquiring a remote sensing image sample dataset, labeling the land cover categories of the remote sensing images in the dataset by pixels, generating corresponding labeled remote sensing images, and automatically generating semantic boundary labels by using an edge detection algorithm on the labeled remote sensing images; After preprocessing, the data is divided into training, validation, and test sets. A remote sensing image semantic segmentation network, including an encoder, a boundary detection branch network, and a boundary point upsampling module, is constructed. The original remote sensing images of the training set are input into the remote sensing image semantic segmentation network for training. The hyperparameters and loss function of the model training are determined, and the parameters of the remote sensing image semantic segmentation network are optimized until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold. The test set is then input into the trained remote sensing image semantic segmentation network to verify the network's segmentation accuracy. The real-time captured remote sensing images are input into the trained remote sensing image semantic segmentation network, and the output result is the semantic segmentation result of the captured remote sensing images. The training method of the remote sensing semantic segmentation network is as follows: the training set of remote sensing images is input into the encoder, the feature maps output by different convolutional layers are input into the boundary detection branch network, the output results of the encoder and the boundary detection branch network are fused to obtain a preliminary segmented remote sensing image, the boundary point upsampling module performs pixel sampling based on the preliminary segmented remote sensing image, and the sampling points, the feature maps output by each convolutional layer of the encoder, and the original remote sensing image input are combined by a small network formed by a multilayer perceptron to make predictions, and the output results are matched and replaced with the preliminary segmented remote sensing image to obtain the boundary-optimized remote sensing image semantic segmentation result.

2. The boundary-optimized semantic segmentation method for remote sensing images according to claim 1, characterized in that, The preprocessing steps for remote sensing images and their corresponding labeled remote sensing images include: Random sampling is performed on the original remote sensing image and the corresponding labeled remote sensing image. Each sampling yields... Remote sensing images of different sizes and their corresponding labels; Sample augmentation is performed on remote sensing images obtained after random sampling, including random rotation. n=0,1,2,3, random horizontal and vertical directions Flip, random scale, random brightness, contrast, and saturation enhancement.

3. The boundary-optimized semantic segmentation method for remote sensing images according to claim 1, characterized in that, The encoder uses ResNet as its backbone network and includes several sequentially connected convolutional layers. The feature map output from the previous convolutional layer is input into the next convolutional layer for feature extraction to generate a new feature map.

4. The boundary-optimized semantic segmentation method for remote sensing images according to claim 3, characterized in that, The boundary detection branch network includes the same number of feature extraction layers as the encoder convolutional layers; each feature extraction layer includes a gated convolution and an upsampling mechanism to control the flow of information; the input of the gated convolution of each feature extraction layer is connected to one of the convolutional layers of the encoder, and the feature map output by the corresponding convolutional layer is used as the gated convolution input of the corresponding feature extraction layer; after the feature map of each convolutional layer is upsampled by the corresponding gated convolution, feature maps of different scales are formed and fused, and then fused with the feature map output by the encoder, resulting in a preliminary segmentation of the remote sensing image.

5. The boundary-optimized semantic segmentation method for remote sensing images according to claim 4, characterized in that, The process of gated convolution is shown in formula (1): （1） Where: X is the input feature map, and i, j represent the position of each pixel. This is the sigmoid function.

6. The boundary-optimized semantic segmentation method for remote sensing images according to claim 4, characterized in that, The boundary point sampling module samples the preliminary segmented remote sensing image and collects pixels with confidence levels lower than a preset threshold as sampling points. The sampling points are matched and fused with feature maps of different scales output by each convolutional layer of the encoder and the original remote sensing image. The fusion result is input into a small network, and the output result is the prediction result of the sampling points. The prediction result of the sampling points is matched and replaced with the initially segmented remote sensing image to obtain the boundary-optimized semantic segmentation result of the remote sensing image.

7. The boundary-optimized semantic segmentation method for remote sensing images according to claim 6, characterized in that, The sampling process of the boundary point sampling module for sampling the initially segmented remote sensing image is as follows: Based on the preliminary segmentation of the remote sensing image, the predicted probability value of each pixel in the image is calculated as shown in formula (2): (2) image Input remote sensing image semantic segmentation network The initial predicted probability map is obtained. K is the number of categories. This represents the pixel in the i-th row and j-th column of X. This represents the predicted probability of the pixel in the i-th row and j-th column of P; the predicted probability map... All pixels are sorted in descending order of probability to obtain the sorted result. ; Calculate the highest probability of classifying all pixels. and the second highest probability The negative of the difference is U This is called uncertainty, as shown in formula (4); （3） （4） This represents the uncertainty of the i-th row and j-th column pixel U. The larger the value, the higher the uncertainty of the prediction result for that pixel. Select N most uncertain pixels, ,remain( Each pixel is sampled uniformly in the image.

8. The boundary-optimized semantic segmentation method for remote sensing images according to claim 6, characterized in that, The steps for sampling point prediction include: Based on the initial segmented remote sensing image, feature reconstruction is performed on the sampled uncertain pixels by combining fine-grained features, as expressed in formula (5): (5) Among them, fine-grained features Low-level feature maps from the encoder and input The encoder is configured with three convolutional layers, and feature maps at three scales are interpolated onto the original remote sensing image to obtain the following results sequentially. , , , The feature maps are fused with the input X to obtain the reconstructed fine-grained feature representation. This indicates a feature map concatenation operation; By concatenating the fine-grained feature G with the predicted probability map P using formula (6), the feature representation of the sampled points reconstructed is obtained. The reconstructed feature representation of each sampling point is extracted based on the coordinate information of the sampled pixels; （6） The reconstructed feature vector is used as input to a small network. The predicted results of the output sampling points are matched and replaced with the initially segmented remote sensing image to obtain the semantic segmentation result of the remote sensing image with optimized boundaries.

9. A boundary-optimized remote sensing image semantic segmentation apparatus, comprising segmenting using the boundary-optimized remote sensing image semantic segmentation method as described in any one of claims 1-8, characterized in that, include: The network construction module is used to construct a remote sensing semantic segmentation network, which is an improvement on the classic semantic segmentation network U-NET. The network training module is used to train the remote sensing semantic segmentation network using a remote sensing image dataset. This includes: acquiring a remote sensing image sample dataset; labeling the land cover categories of the remote sensing images in the dataset by pixels to generate corresponding labeled remote sensing images; automatically generating semantic boundary labels by using an edge detection algorithm on the labeled remote sensing images; dividing the preprocessed dataset into training, validation, and test sets; constructing a remote sensing image semantic segmentation network including an encoder, a boundary detection branch network, and a boundary point upsampling module; inputting the original remote sensing images from the training set into the remote sensing image semantic segmentation network for training; determining the hyperparameters and loss function for model training; optimizing the parameters of the remote sensing image semantic segmentation network until the prediction accuracy of the validation set prediction results reaches a preset accuracy threshold; and inputting the test set into the trained remote sensing image semantic segmentation network to verify the network's segmentation accuracy. The image segmentation module is used to input real-time captured remote sensing images into a trained remote sensing image semantic segmentation network. The output result is the semantic segmentation result of the captured remote sensing images. The training method of the remote sensing semantic segmentation network is as follows: the training set of remote sensing images is input into the encoder, the feature maps output from different convolutional layers are input into the boundary detection branch network, and the output results of the encoder and the boundary detection branch network are fused to obtain a preliminary segmented remote sensing image. The boundary point upsampling module performs pixel sampling based on the preliminary segmented remote sensing image, and predicts the sampling points, the feature maps output from each convolutional layer of the encoder, and the original remote sensing image input by a small network formed by combining multilayer perceptrons. The output result is matched and replaced with the preliminary segmented remote sensing image to obtain the boundary-optimized remote sensing image semantic segmentation result.

10. The boundary-optimized remote sensing image semantic segmentation apparatus according to claim 9, characterized in that, The hyperparameters for model training include using the Adam optimizer, exponential decay rate, regularization coefficient, or learning rate. Since it involves two tasks, semantic segmentation and boundary detection, and also requires training a small network in the boundary point upsampling module, the overall loss function consists of three parts, as shown in Equation (7). The semantic segmentation task and the small network are trained using the cross-entropy loss function, while the boundary detection branch network is trained using the binary cross-entropy loss function. (7) and These represent the parameters of the semantic segmentation backbone network and the boundary detection branch network, respectively. Parameters representing small networks, For semantic boundary labels, The prediction results of the boundary detection branch network, For semantic segmentation tags, The prediction results are from the semantic segmentation network. Labels for uncertain points sampled by the boundary point sampling module. For prediction results of small networks, , It is a hyperparameter that adjusts the weight of semantic segmentation loss and boundary extraction loss.

11. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images