Remote sensing image semantic segmentation model training method and remote sensing image semantic segmentation model

Abstract

The application discloses a remote sensing image semantic segmentation model training method and a remote sensing image semantic segmentation model. The method comprises the following steps: performing L sets of first processing on a remote sensing image to obtain L first processing results; performing up-sampling processing on the last first processing result; performing processing on P1 third features and / or P2 fifth features and a remote sensing label image respectively to obtain P1 fourth features and / or P2 sets of predicted global frequency distributions and a set of real global frequency distributions, which are used to determine P1 first losses and / or P2 second losses; extracting spatial detail features of labels in the remote sensing label image to obtain sixth features according to the labels; the P1 third features and the P2 fifth features respectively comprise at least one of the first processing results and / or the up-sampling processing results; determining a third loss according to a class probability of a pixel in the last up-sampling processing result and the remote sensing label image, and determining a fourth loss according to the third loss, the P1 first losses and / or the P2 second losses.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image semantic segmentation. In particular, it relates to a remote sensing image semantic segmentation model training method and a remote sensing image semantic segmentation model. BACKGROUND

[0002] With the rapid development of earth observation and remote sensing imaging technology, remote sensing has entered the era of big data. The characteristics of remote sensing big data mainly include three aspects: the quantity, download speed and types of data. In the era of remote sensing big data, a large amount of remote sensing data needs to be processed every day. In addition, more and more different remote sensing data plays an important role in different fields. Due to the breakthrough of imaging technology, very high resolution (VHR) remote sensing images (RSI) show great potential in RSI interpretation and become the focus of semantic segmentation. Semantic segmentation needs to pixelize the input remote sensing image to obtain the category to which each pixel belongs.

[0003] The specific application of semantic segmentation in the field of remote sensing is, for example, comprehensive land use and land cover mapping, urban change detection, urban functional zones, building footprints, impervious surfaces and water body extraction. Most of these applications are based on VHR remote sensing images and are limited by the following two problems. 1) Low-detail information modeling. Compared with previous low-resolution remote sensing images, VHR remote sensing images provide unbalanced spatial detail information and improved semantic information. The significant increase in spatial resolution enables people to observe spatial detail information that was previously unable to be captured. However, at the same time, key spatial detail information is mixed in a large amount of redundant information, which brings new challenges to information extraction. 2) Low processing efficiency. On the data processing side, VHR remote sensing images mean that the amount of interpretation data to be processed per unit of observation area increases greatly, which is undoubtedly a major challenge to hardware and algorithms.

[0004] To address the semantic segmentation problem of VHR remote sensing images in the era of big data, researchers have proposed several methods. Deep learning algorithms are currently the mainstream approach for semantic segmentation. Unlike traditional machine learning algorithms based on prior knowledge and predefined rules, deep learning algorithms are data-driven and perform poorly on small datasets, but can play a significant role in the era of big data. Convolutional neural networks (CNNs) based on deep learning have demonstrated outstanding performance, surpassing that of traditional machine learning algorithms. Fully convolutional networks (FCNs) have been used for semantic segmentation of RSI and have achieved impressive results. Subsequent research has led to the development of many new model variants based on the FCN architecture, achieving significant progress in various aspects. Based on an encoder-decoder architecture, UNet enhances the ability of FCNs to represent multi-scale features of images by shrinking and expanding paths to achieve high-precision road and coastline recognition in RSI. The DeepLabv3 series utilizes atrous spatial pyramid pooling (ASPP) with varying proportions to increase the model's receptive field while acquiring multi-scale features; these models are widely used for RSI semantic segmentation, cloud detection, and other applications. However, these models are slow inference and have high hardware requirements for deployment devices, making it difficult for these methods to solve both of the above problems at the same time.

[0005] Therefore, how to improve the accuracy and inference speed of semantic segmentation of VHR remote sensing images has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] Due to the aforementioned problems with existing methods, this application proposes a training method and a semantic segmentation model for remote sensing images.

[0007] Firstly, this application proposes a method for training a semantic segmentation model for remote sensing images, including:

[0008] Acquire remote sensing images and their corresponding remote sensing label images, wherein the labels in the remote sensing label images are used to characterize the category of each pixel;

[0009] The remote sensing image is subjected to L groups of first processing to obtain L first processing results; wherein the last first processing result is the first feature. Each group of first processing in the L groups includes at least one convolution processing, and the total number of convolution processing in all groups of first processing is M. The stride of the convolution kernel used in M1 of the M convolution processing is greater than 1. L is a positive integer, M is a positive integer, and M1 is a positive integer greater than 1 and less than M.

[0010] The first feature is upsampled L1 times to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; the last upsampling result is the second feature.

[0011] Feature selection processing is performed on P1 third features to obtain P1 fourth features, where P1 is a positive integer. The P1 third features include at least one of L first processing results and / or L1 upsampling processing results.

[0012] Based on the labels, spatial detail features of the remote sensing label images are extracted to obtain the sixth feature;

[0013] Based on the P1 fourth features and the sixth feature, determine the P1 first losses; and / or

[0014] Hierarchical semantic boundary segmentation is performed on P2 fifth features and the remote sensing label image to obtain P2 sets of predicted global frequency distributions and a set of real global frequency distributions, where P2 is a positive integer. The P2 fifth features include at least one of L first processing results and / or L1 upsampling processing results.

[0015] Based on the P2 group of predicted global frequency distributions and the group of actual global frequency distributions, determine P2 second losses;

[0016] Classify each pixel in the second feature to determine the class probability of each pixel in the second feature;

[0017] The third loss is determined based on the class probability of each pixel in the second feature and the class probability of each pixel in the remote sensing label image;

[0018] The fourth loss is determined based on the P1 first loss, the P2 second loss, and / or the third loss;

[0019] Based on the fourth loss, the parameters of the remote sensing image semantic segmentation model are determined;

[0020] Repeat the above steps until the termination condition for training the remote sensing image semantic segmentation model is met.

[0021] In one possible implementation, the method further includes:

[0022] The result of each upsampling process is added pixel by pixel to the first processing result with the same resolution to obtain the input for the next upsampling process.

[0023] In one possible implementation, the step of performing feature selection processing on P1 third features to obtain P1 fourth features includes:

[0024] For each third feature, features from at least one channel are selected in the same proportion to obtain P1 fifth features;

[0025] The P1 fifth features are subjected to a first convolution process to obtain P1 fourth features.

[0026] In one possible implementation, extracting spatial detail features of the remote sensing tag image based on the tag to obtain a sixth feature includes:

[0027] Based on the label, at least one seventh feature is obtained by extracting spatial detail features of the remote sensing label image using at least one Laplacian convolution operator with different strides.

[0028] The sixth feature is obtained based on the at least one seventh feature.

[0029] In one possible implementation, the hierarchical semantic boundary segmentation processing of the P2 fifth features and the remote sensing label image to obtain P2 sets of predicted global frequency distributions and a set of true global frequency distributions includes:

[0030] A hierarchical semantic boundary is set for the remote sensing tag image, and the semantic boundary has a total boundary level of S, where S is an integer greater than or equal to 0;

[0031] Cut the semantic boundary at boundary level s along its length and width by 2. s 2 times, obtained 2 2s There are 3 labeled image blocks; where s is greater than or equal to 0 and less than or equal to S. It is a set of labeled image patches within a semantic boundary of boundary level s, and

[0032] Confirm 2 2s The first local frequency distribution, the 2 2s The first local frequency distributions are 2 2s The true local frequency distribution of pixels of each category in each labeled image patch;

[0033] According to the above 2 2s A first local frequency distribution is used to determine the set of true global frequency distributions;

[0034] Repeat the above steps for P2 fifth features to obtain the predicted global frequency distribution of P2 groups.

[0035] Secondly, this application also proposes a remote sensing image semantic segmentation model, including:

[0036] An encoder is used to acquire remote sensing images; L groups of first processing are performed on the remote sensing images to obtain L first processing results; wherein the last first processing result is a first feature; each group of first processing in the L groups of first processing includes at least one convolutional processing, and the total number of convolutional processing in all groups of first processing is M, the stride of the convolutional kernel used in M1 of the M convolutional processing is greater than 1, L is a positive integer, M is a positive integer, and M1 is a positive integer greater than 1 and less than M to obtain L downsampling processing results with successively decreasing resolution, where L is a positive integer; wherein the last downsampling processing result is a first feature;

[0037] The decoder is used to perform L1 upsampling processes on the first feature to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; the last upsampling result is the second feature; and the second feature is used to classify each pixel to determine the class probability of each pixel in the second feature.

[0038] A multiscale spatial detail (MSD) enhancement unit is used to perform feature selection processing on P1 third features to obtain P1 fourth features; wherein, the P1 third features include at least one of L first processing results and / or L1 upsampling processing results; a remote sensing label image corresponding to the remote sensing image is obtained, wherein the labels in the remote sensing label image are used to characterize the category of each pixel therein; based on the labels, the spatial detail features of the remote sensing label image are extracted to obtain a sixth feature;

[0039] The hierarchical semantic enhancement (HSE) unit is used to acquire remote sensing label images corresponding to remote sensing images; hierarchical semantic boundary segmentation processing is performed on P2 fifth features and the remote sensing label images respectively to obtain P2 sets of predicted global frequency distributions and a set of true global frequency distributions, where P2 is a positive integer, and the P2 fifth features include at least one of L first processing results and / or L1 upsampling processing results.

[0040] In one possible implementation, the multi-scale spatial detail (MSD) enhancement unit is specifically used to select features from at least one channel of each third feature in the same proportion to obtain P1 fifth features; and to perform convolution processing on the P1 fifth features to obtain P1 fourth features.

[0041] In one possible implementation, the hierarchical semantic enhancement (HSE) unit is specifically used to set hierarchical semantic boundaries for the remote sensing label image. These hierarchical semantic boundaries have a total boundary level of S, where S is an integer greater than or equal to 0. The semantic boundary at boundary level s is then diced along its length and width by 2... s 2 times, obtained 2 2s There are 3 labeled image blocks; where s is greater than or equal to 0 and less than or equal to S. It is a set of labeled image patches within a semantic boundary of boundary level s, and

[0042] Confirm 2 2s The first local frequency distribution, the 2 2s The first local frequency distributions are 2 2s The true local frequency distribution of pixels of each category in each label image block; according to the 2 2s A first local frequency distribution is used to determine the set of true global frequency distributions; the aforementioned steps are repeated for P2 fifth features to obtain P2 sets of predicted global frequency distributions.

[0043] Thirdly, this application also proposes an electronic device including at least one processor for executing a program stored in a memory, which, when executed, causes the device to perform various steps as described in the first aspect and various possible implementations.

[0044] Fourthly, this application also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the first aspect and various possible implementations. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the structure of the remote sensing image semantic segmentation model provided in the embodiments of this application;

[0047] Figure 2 A schematic diagram illustrating the operation of the embedding attention module (EAM) provided in an embodiment of this application;

[0048] Figure 3This is another structural schematic diagram of the remote sensing image semantic segmentation model provided in the embodiments of this application;

[0049] Figure 4 A flowchart illustrating the training method for a remote sensing image semantic segmentation model provided in this application embodiment;

[0050] Figure 5 A schematic diagram of an asymmetric, lightweight, U-shaped CNN provided for embodiments of this application;

[0051] Figure 6 This is a schematic diagram of the operation of the MSD enhancement unit provided in the embodiments of this application;

[0052] Figure 7 This is a schematic diagram of the HSE unit providing an embodiment of this application performing hierarchical semantic boundary segmentation processing on a remote sensing tag image;

[0053] Figure 8 A schematic diagram illustrating the inference speed and accuracy (intersection over union, LOU) results obtained by applying the technical solution provided in this application to the Potsdam and Vaihingen datasets, as provided in the embodiments of this application.

[0054] Figure 9 A schematic diagram illustrating the effectiveness of the EAM, MSD enhancement unit, and HSE unit provided in the embodiments of this application for semantic segmentation of remote sensing images;

[0055] Figure 10 and Figure 11 This is a schematic diagram illustrating the semantic segmentation results of Potsdam data for small-sized and large-sized remote sensing images, as provided in the embodiments of this application.

[0056] Figure 12 and Figure 13 This diagram illustrates the semantic segmentation results of the Faiingen dataset, which demonstrates the semantic segmentation performance of small-sized and large-sized remote sensing images in this embodiment of the application. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0058] It should be noted that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The terms "first" to "seventh," etc., in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature," "second feature," and "second feature," etc., are used to distinguish different features, not to describe a specific order of target objects. In the embodiments of this application, words such as "exemplary," "for example," or "e.g.," are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary," "for example," or "e.g.," in the embodiments of this application should not be construed as superior to other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "e.g.," is intended to present related concepts in a specific manner. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0059] To address the low accuracy and slow inference speed of current VHR remote sensing image semantic segmentation, this application proposes a training method for a remote sensing image semantic segmentation model. This training method is based on the remote sensing image semantic segmentation model proposed in this application. A schematic diagram of the structure of the remote sensing image semantic segmentation model is shown in Figure 100. Figure 1 As shown in the diagram. The structural schematic 100 includes an encoder 101, a decoder 102, a multi-scale spatial detail (MSD) enhancement unit 103, and a hierarchical semantic enhancement (HSE) unit 104.

[0060] The encoder 101 is used to acquire remote sensing images; L groups of first processing are performed on the remote sensing images to obtain L first processing results; wherein, the last first processing result is a first feature, each group of first processing in the L groups of first processing includes at least one convolution processing, and the total number of convolution processing included in all groups of first processing is M, the stride of the convolution kernel used in M1 of the M convolution processing is greater than 1, L is a positive integer, M is a positive integer, and M1 is a positive integer greater than 1 and less than M.

[0061] Decoder 102 is connected to encoder 101. Decoder 102 performs L1 upsampling processes on the first feature to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; the last upsampling result is the second feature; and it classifies each pixel in the second feature to determine the class probability of each pixel. It should be noted that in this embodiment, decoder 102 adds each upsampling result to the first processing result with the same resolution pixel by pixel to obtain the input for the next upsampling process.

[0062] The multi-scale spatial detail (MSD) enhancement unit 103 is connected to both the decoder 101 and the encoder 102. The multi-scale spatial detail (MSD) enhancement unit 103 performs feature selection processing on P1 third features to obtain P1 fourth features; wherein the P1 third features include at least one of L first processing results and / or L1 upsampling processing results; it acquires a remote sensing label image corresponding to the remote sensing image, where the label (ground truth, GT) in the remote sensing label image is used to characterize the category of each pixel; based on the label, it extracts spatial detail features from the remote sensing label image to obtain a sixth feature.

[0063] In one possible implementation, the multi-scale spatial detail (MSD) enhancement unit 103 is used to perform feature selection processing on P1 third features to obtain P1 fourth features. The specific process is as follows:

[0064] First, features of at least one channel of each third feature are selected in the same proportion to obtain P1 fifth features;

[0065] Then, the P1 fifth features are subjected to the first convolution process to obtain the P1 fourth features.

[0066] In one possible implementation, the multi-scale spatial detail (MSD) enhancement unit 103 extracts spatial detail features from the remote sensing labeled image based on the label to obtain the sixth feature. The specific process is as follows:

[0067] First, based on the label, spatial detail features of the remote sensing label image are extracted using at least one Laplacian convolution operator with different strides to obtain at least one seventh feature;

[0068] Then, the sixth feature is obtained based on the at least one seventh feature.

[0069] The hierarchical semantic enhancement (HSE) unit 104 is connected to both the decoder 101 and the encoder 102. The HSE unit 104 is used to acquire remote sensing label images corresponding to the remote sensing images; it performs hierarchical semantic boundary segmentation processing on P2 fifth features and the remote sensing label images respectively, obtaining P2 sets of predicted global frequency distributions and one set of true global frequency distributions, where P2 is a positive integer. The P2 fifth features include at least one of L downsampling processing results and / or L1 upsampling processing results.

[0070] In one possible implementation, the hierarchical semantic enhancement (HSE) unit 104 is used to perform hierarchical semantic boundary segmentation on P2 fifth feature and remote sensing label images respectively, and to obtain P2 sets of predicted global frequency distributions and a set of true global frequency distributions. The specific process is as follows:

[0071] First, hierarchical semantic boundaries are set for the remote sensing tag image. The semantic boundaries have a total boundary level of S, where S is an integer greater than or equal to 0.

[0072] Secondly, the semantic boundary of the boundary level s is cut along its length and width by 2. s 2 times, obtained 2 2s 1 label image block; among which It is a set of labeled image patches within a semantic boundary of boundary level s, and

[0073] Furthermore, confirm 2 2s The first local frequency distribution, the 2 2s The first local frequency distributions are 2 2s The true local frequency distribution of pixels of each category in each labeled image patch;

[0074] Then, according to the 2 2s A first local frequency distribution is used to determine the set of true global frequency distributions;

[0075] Finally, repeat the above steps for P2 fifth features to obtain the predicted global frequency distribution of P2 groups.

[0076] It is also understandable that, due to the large scale differences and complex backgrounds of remote sensing images, they possess characteristics such as... Figure 1 The remote sensing image semantic segmentation models with the structure shown often struggle to enhance the representation of key features. This is typically addressed by embedding an attention module (EAM), but this approach is complex and fails to meet the efficiency requirements of remote sensing image semantic segmentation. Therefore, this application proposes a simple EAM105 to generate corresponding attention weights and improve the representation of key features, as shown in the attached diagram. Figure 2 As shown in Figure 300, EAM105 can be located between encoder 101 and decoder 102. Figure 3 As shown. By increasing the weights of important regions and suppressing background regions, the features are adaptively calibrated in the spatial dimension, which can more effectively improve classification performance, thereby improving the semantic segmentation accuracy of remote sensing images. At this point, the last downsampling result, i.e., the first feature X, is input into EAM105 and processed as follows:

[0077] First, EAM105 processes the first feature X to obtain the eighth feature. Specifically, the first feature X is multiplied by a preset parameter matrix to generate the query matrix. The key matrix K and the value matrix V have dimensions respectively. and Where d k =d vThe key matrix K and value matrix V are completely independent of the first feature X, are randomly generated during initialization, and are retained in subsequent calculations. First, the query matrix Q and the key matrix K are multiplied to obtain A. i，j Attention score map is calculated using the softmax function. The calculation formula is as follows:

[0078]

[0079]

[0080] Since the key matrix K and value matrix V are completely independent of the first feature X, the interaction between elements is weak, making it difficult for EAM to learn truly effective attentional relationships. Using only a single softmax function to strengthen the connections between elements, which is common in EAM, may not yield good results. Therefore, a method was adopted... The final attention map is obtained by performing L1 normalization on both rows and columns. The calculation formula is as follows:

[0081]

[0082] Ultimately, A score Attention map Multiplying the value matrix V by the matrix yields the final output X. sa That is, the eighth characteristic:

[0083]

[0084] Then, decoder 102 performs L1 upsampling on the eighth feature to obtain the second feature. Subsequent processing of the second feature follows the same procedure as described above for decoder 102.

[0085] This application embodiment improves the accuracy of remote sensing image semantic segmentation by adding MSD enhancement unit 103 and HSE unit 104. Adding EAM 105 also improves the accuracy of remote sensing image semantic segmentation. The simplified remote sensing image semantic segmentation model accelerates inference speed.

[0086] Next, the training method for the remote sensing image semantic segmentation model proposed in this application will be described in detail. This training method is based on the aforementioned remote sensing image semantic segmentation model. Its flowchart is shown below. Figure 4 As shown in the diagram. This process diagram includes: S401-S413.

[0087] First, it should be noted that the encoder 101, decoder 102, MSD enhancement unit 103, and HSE unit 104 are used in the training of the remote sensing image semantic segmentation model. When determining the loss, units other than those in the remote sensing image semantic segmentation model 300 need to be used; this unit is referred to as the first unit.

[0088] In one possible implementation, the remote sensing image semantic segmentation model training method provided in this application embodiment is achieved through the following steps:

[0089] S401, encoder 101 acquires remote sensing images.

[0090] In this embodiment, encoder 101 acquires a remote sensing image. This remote sensing image can be a VHR remote sensing image.

[0091] S402, encoder 101 performs L groups of first processing on the remote sensing image to obtain L first processing results.

[0092] In this embodiment of the application, encoder 101 performs L sets of first processing on remote sensing images to obtain L first processing results, where L is a positive integer.

[0093] For example, the semantic segmentation model for remote sensing images can be a CNN. Considering that typical semantic segmentation tasks only downsample remote sensing images to a 1 / 16 or 1 / 32 resolution scale via an encoder, this is completely insufficient for VHR remote sensing images. Therefore, embodiments of this application can train a CNN as follows: Figure 5 The asymmetric, lightweight, U-shaped CNN shown here has an exemplary encoder structure shown in Table 1. The trained CNN can downsample VHR remote sensing images to 1 / 64.

[0094] Table 1

[0095]

[0096] In one example, L takes the value 7. For example... Figure 5As shown, after the first processing (Stage 0), the resolution of a 512×512 pixel remote sensing image is reduced to 1 / 4, resulting in a 128×128 pixel image. After the second processing (Stage 1), the resolution remains 1 / 4, resulting in a 128×128 pixel image. After the third processing (Stage 2), the resolution is reduced to 1 / 8, resulting in a 64×64 pixel image. After the fourth processing (Stage 3), the resolution remains 1 / 8, resulting in a 64×64 pixel image. After the fourth processing step (Stage 4), the resolution of the 64×64 pixel result is reduced to 1 / 16, resulting in a 32×32 pixel result. After the sixth processing step (Stage 5), the resolution is reduced to 1 / 32, resulting in a 16×16 pixel result. After the seventh processing step (Stage 6), the resolution is reduced to 1 / 64, resulting in an 8×8 pixel result.

[0097] S403, decoder 102 performs L1 upsampling processing on the first feature to obtain L1 upsampling processing results with progressively increasing resolution.

[0098] In this embodiment, the decoder 102 performs L1 upsampling processes on the first feature to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; and the last upsampling result is the second feature.

[0099] In one possible implementation, decoder 102 adds the result of each upsampling process to the first processing result with the same resolution pixel by pixel to obtain the input for the next upsampling process.

[0100] In one example, L1 takes the value 4. For example... Figure 5As shown, the MSD enhancement unit 103 includes Aux Head1, Aux Head2, Aux Head3, Aux Head4, and Aux Head6, the HSE unit 104 includes Aux Head7, the encoder 101 is shown within the dashed box 501, and the decoder 102 is shown within the dashed box 502. After the first upsampling process on the 8×8 pixel first result, a 16×16 pixel upsampling result is obtained. The 16×16 pixel upsampling result and the 16×16 pixel first result are added pixel by pixel to obtain the input for the second upsampling process. After the second upsampling process on this input, a 32×32 pixel upsampling result is obtained. The 32×32 pixel upsampling result and the 32×32 pixel first result are added pixel by pixel to obtain the input for the third upsampling process. After the third upsampling process on this input, a 64×64 pixel upsampling result is obtained. The upsampling result of 64×64 pixels and the first processing result of the fourth processing of 64×64 pixels are added pixel by pixel to obtain the input for the fourth upsampling processing. After performing the fourth upsampling processing on this input, the upsampling result of 128×128 pixels is obtained.

[0101] It should be noted that, in this embodiment, to effectively utilize the detailed information of VHR remote sensing images, widely used detail enhancement methods include spatial attention modules. However, these methods tend to introduce excessive computational overhead, resulting in low semantic segmentation efficiency. This embodiment designs a selection kernel-based MSD enhancement unit 103 and a loss function. The operations performed by the selection kernel-based MSD enhancement unit 103 include: S404 and S405.

[0102] S404, MSD enhancement unit 103 performs feature selection processing on P1 third features to obtain P1 fourth features.

[0103] In this embodiment, the MSD enhancement unit 103 performs feature selection processing on P1 third features to obtain P1 fourth features, where P1 is a positive integer. The P1 third features include at least one of L first processing results and / or L1 upsampling processing results. The third feature is represented as F... in ∈R C×H×W , where R is the set of real numbers, C is the channel dimension of the third feature, H is the length of the third feature, and W is the width of the third feature.

[0104] In one possible implementation, the MSD enhancement unit 103 selects features of at least one channel of each third feature in the same proportion, for F inThe channel dimension is truncated to obtain P1 fifth features. The MSD enhancement unit 103 performs a first convolution process on each of the P1 fifth features to obtain P1 fourth features, denoted as F. S ∈R C×H×W In one possible implementation, the first convolutional processing includes: convolution using a 3×3 convolution kernel, batch normalization, processing with a ReLU activation function, and convolution using a 1×1 convolution kernel.

[0105] In one example, such as Figure 6 As shown, the six-channel features are called the third feature. After the six-channel features are processed by a proportional feature selection kernel, i.e., randomly selected at a 1 / 2 ratio, three-channel features are obtained, which are called the fifth feature. The MSD enhancement unit 103 performs convolution on the fifth feature using a 3×3 convolution kernel, batch normalization, ReLU activation function, and a 1×1 convolution kernel to obtain the fourth feature. The fourth feature has a channel dimension of 1 and matches the shape of the detail feature map pyramid.

[0106] S405, MSD enhancement unit 103 acquires the remote sensing tag image corresponding to the remote sensing image, and extracts the spatial detail features of the remote sensing tag image based on the tags in the remote sensing tag image to obtain the sixth feature.

[0107] In this embodiment of the application, the MSD enhancement unit 103 acquires a remote sensing tag image corresponding to the remote sensing image, and extracts the spatial detail features of the remote sensing tag image based on the tag in the remote sensing tag image to obtain the sixth feature.

[0108] In one possible implementation, the MSD enhancement unit 103 extracts spatial detail features of the remote sensing labeled image using at least one Laplacian convolution operator with different strides, based on the label, to obtain at least one seventh feature. The MSD enhancement unit 103 then obtains the sixth feature based on the at least one seventh feature.

[0109] It is understood that the embodiments of this application construct a multi-scale edge extraction pyramid. The most frequently used second-order differential operator is the two-dimensional Laplacian operator, which is described as follows:

[0110]

[0111] Here, Δf is a real-valued function that is twice differentiable. To handle RSI in discrete data form, the discrete Laplace operator is applied, as shown in the following equation:

[0112]

[0113] For example, such as Figure 6As shown, the MSD enhancement unit 103 extracts spatial detail features from the remote sensing labeled image using three Laplacian convolution operators based on the label, obtaining three seventh features, namely D0, D2, and D4∈R. H×w The MSD enhancement unit 103 obtains a sixth feature based on the three seventh features, where the sixth feature P∈R. H×W :

[0114]

[0115] For example, such as Figure 6 As shown, the three seventh features are added pixel by pixel to obtain the detail feature map pyramid, which is the sixth feature.

[0116] The embodiments of this application can improve the accuracy of semantic segmentation of remote sensing images by adding the MSD enhancement unit 103.

[0117] S406, the first unit determines the first loss P1 based on the fourth and sixth features P1.

[0118] In this embodiment of the application, the first unit determines P1 first losses based on P1 fourth features and sixth features.

[0119] It is understandable that, considering the extremely small proportion of spatial detail features (i.e., edge and texture information) in the entire remote sensing image, it is difficult to obtain good results using only the binary cross-entropy (BCE) loss function for a sparse matrix like the detail feature map pyramid P. Therefore, this application's embodiments optimize the loss evaluation method by introducing a dice loss function that is insensitive to class proportions. This dice loss has reliable discriminative ability between foreground and background information. The formulas for BCE loss and DICE loss are as follows:

[0120]

[0121]

[0122] Among them, f i and p i Representing from F S The value of the i-th element obtained from P is given by ε, which is a very small number used to smooth the gradient and is set to 1e-8. The first loss is obtained by adding the BCE loss and the DICE loss. Accordingly, P1 first losses can be obtained by formulas (8) and (9).

[0123] S407, HSE unit 104 acquires the remote sensing label image corresponding to the remote sensing image, and performs hierarchical semantic boundary segmentation processing on P2 fifth features and remote sensing label images respectively, to obtain P2 sets of predicted global frequency distributions and a set of real global frequency distributions respectively.

[0124] In this embodiment of the application, the hierarchical semantic enhancement (HSE) unit 104 acquires the remote sensing label image corresponding to the remote sensing image, and performs hierarchical semantic boundary segmentation processing on P2 fifth features and the remote sensing label image respectively to obtain P2 sets of predicted global frequency distributions and a set of real global frequency distributions, where P2 is a positive integer. The P2 fifth features include at least one of L first processing results and / or L1 upsampling processing results.

[0125] In one possible implementation, it is understood that because pixels in VHR remote sensing images are divided into many categories, these categories often have significant intra-class spectral differences and small inter-class spectral differences, leading to category resolution errors. The embodiments of this application can effectively mitigate the impact of this problem on the resulting segmentation performance. Connecting the proposed HSE unit 104 to the decoder incurs no additional inference cost. The semantic frequency distribution-based HSE unit 104 and the loss function are described in detail below. The HSE unit 104 sets hierarchical semantic boundaries for the remote sensing label image, where the semantic boundaries have a common boundary level S, where S is an integer greater than or equal to 0; the HSE unit 104 divides the semantic boundaries of boundary level s along a length and width of 2... s 2 times, obtained 2 2s 1 label image block; among which It is a set of labeled image patches within a semantic boundary of boundary level s, and HSE Unit 104 Determined 2 2s The first local frequency distribution, the 2 2s The first local frequency distributions are 2 2s The true local frequency distribution of pixels of each category in each label image block; HSE unit 104 according to the 2 2s The first local frequency distribution is used to determine the set of true global frequency distributions; the HSE unit 104 repeats the aforementioned steps for the P2 fifth features to obtain the P2 set of predicted global frequency distributions.

[0126] For example, such as Figure 7 As shown, S takes the value 3. The semantic boundary of layer 0 (the top layer) corresponds to 1 first local frequency distribution, the semantic boundary of layer 1 corresponds to 4 first local frequency distributions, the semantic boundary of layer 2 corresponds to 16 first local frequency distributions, and the semantic boundary of layer 3 corresponds to 64 first local frequency distributions. Connecting the first local frequency distributions of the semantic boundaries of each layer, as shown...Figure 7 As shown, a set of true global frequency distributions is obtained.

[0127] The embodiments of this application can improve the accuracy of semantic segmentation of remote sensing images by adding HSE unit 104.

[0128] In one possible implementation, the aforementioned steps are repeated for P2 fifth features to obtain P2 sets of predicted global frequency distributions.

[0129] S408, the first unit determines P2 second losses based on the predicted global frequency distribution of P2 groups and a set of real global frequency distributions.

[0130] In this embodiment of the application, the first unit determines P2 second losses based on the P2 group of predicted global frequency distributions and a group of real global frequency distributions.

[0131] In one example, the first unit can determine P2 second losses based on the P2 predicted global frequency distributions, a set of true global frequency distributions, and the BCE loss function.

[0132] S409, Decoder 102 classifies each pixel in the second feature and determines the class probability of each pixel in the second feature.

[0133] In this embodiment of the application, the decoder 102 classifies each pixel in the second feature and determines the category probability of each pixel in the second feature.

[0134] In one possible implementation, embodiments of this application define the pixel category as impermeable surface, building, low vegetation, tree, car, and background, etc. Correspondingly, the category probability of each pixel in the second feature is the probability that each pixel belongs to one of the six categories: impermeable surface, building, low vegetation, tree, car, and background.

[0135] S410, the first unit determines the third loss based on the class probability of each pixel in the second feature and the class probability of each pixel in the remote sensing label image.

[0136] In this embodiment of the application, the first unit determines the third loss based on the class probability of each pixel in the second feature and the class probability of each pixel in the remote sensing tag image.

[0137] S411, the first unit determines the fourth loss based on P1 first losses, P2 second losses and / or third losses.

[0138] In this embodiment of the application, the first unit determines the fourth loss based on P1 first losses, P2 second losses and / or the third loss.

[0139] In one possible implementation, the first unit adds P1 first losses, P2 second losses, and the third loss to obtain the fourth loss.

[0140] S412, the first unit determines the parameters of the remote sensing image semantic segmentation model based on the fourth loss.

[0141] In this embodiment of the application, the first unit determines the parameters of the remote sensing image semantic segmentation model based on the fourth loss.

[0142] S413, Repeat the above steps until the termination condition for training the remote sensing image semantic segmentation model is met.

[0143] In this embodiment of the application, the above steps are repeated until the termination condition for training the remote sensing image semantic segmentation model is met.

[0144] This application embodiment improves the accuracy of remote sensing image semantic segmentation by adding MSD enhancement unit 103 and HSE unit 104. Adding EAM also improves the accuracy of remote sensing image semantic segmentation. The simplified remote sensing image semantic segmentation model accelerates inference speed.

[0145] The embodiments of this application have been validated on the Potsdam and Vaihingen datasets, and the inference speed and accuracy results are attached. Figure 8 As shown, a better trade-off between inference speed and accuracy can be achieved; the effectiveness of the proposed EAM105, MSD enhancement unit 103, and HSE unit 104 in semantic segmentation of remote sensing images is shown in the appendix. Figure 9 The semantic segmentation results for the Potsdam and Faingen datasets are shown in Tables 3 and 4, respectively. The semantic segmentation results for small and large remote sensing images are also shown in the appendix. Figure 10 and attached Figure 11 As shown, the results are quite good. The semantic segmentation results of the Faiingen dataset are attached. Figure 12 and attached Figure 13 As shown.

[0146] Table 2

[0147]

[0148]

[0149] Table 3

[0150]

[0151] Table 4

[0152]

[0153]

[0154] This application provides an electronic device including at least one processor, the processor being configured to execute a program stored in a memory, wherein when the program is executed, the device performs actions such as... Figure 4 The steps shown.

[0155] This application provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the following... Figure 4 The steps shown.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] It should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a semantic segmentation model for remote sensing images, characterized in that, include: Acquire remote sensing images and their corresponding remote sensing label images, wherein the labels in the remote sensing label images are used to characterize the category of each pixel; The remote sensing image is subjected to L groups of first processing to obtain L first processing results; wherein the last first processing result is the first feature. Each group of first processing in the L groups includes at least one convolution processing, and the total number of convolution processing in all groups of first processing is M. The stride of the convolution kernel used in M1 of the M convolution processing is greater than 1. L is a positive integer, M is a positive integer, and M1 is a positive integer greater than 1 and less than M. The first feature is upsampled L1 times to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; the last upsampling result is the second feature. Feature selection processing is performed on P1 third features to obtain P1 fourth features, where P1 is a positive integer. The P1 third features include at least one of L first processing results and L1 upsampling processing results. Based on the labels, spatial detail features of the remote sensing label images are extracted to obtain the sixth feature; Based on the P1 fourth features and the sixth feature, determine the P1 first losses; Hierarchical semantic boundary segmentation is performed on P2 fifth features and the remote sensing label image to obtain P2 sets of predicted global frequency distributions and one set of real global frequency distributions, where P2 is a positive integer. The P2 fifth features include at least one of L first processing results and L1 upsampling processing results. Based on the P2 group of predicted global frequency distributions and the group of actual global frequency distributions, determine P2 second losses; Classify each pixel in the second feature to determine the class probability of each pixel in the second feature; The third loss is determined based on the class probability of each pixel in the second feature and the class probability of each pixel in the remote sensing label image; The fourth loss is determined based on the P1 first loss, the P2 second loss, and the third loss; Based on the fourth loss, the parameters of the remote sensing image semantic segmentation model are determined; Repeat the above steps until the termination condition for training the remote sensing image semantic segmentation model is met.

2. The method according to claim 1, characterized in that, The method further includes: The result of each upsampling process is added pixel by pixel to the first processing result with the same resolution to obtain the input for the next upsampling process.

3. The method according to claim 1, characterized in that, The step of performing feature selection processing on P1 third features to obtain P1 fourth features includes: For each third feature, features from at least one channel are selected in the same proportion to obtain P1 fifth features; The P1 fifth features are subjected to a first convolution process to obtain P1 fourth features.

4. The method according to claim 1, characterized in that, The step of extracting spatial detail features from the remote sensing tag image based on the tag to obtain a sixth feature includes: Based on the label, at least one seventh feature is obtained by extracting spatial detail features of the remote sensing label image using at least one Laplacian convolution operator with different strides. The sixth feature is obtained based on the at least one seventh feature.

5. The method according to claim 1, characterized in that, The hierarchical semantic boundary segmentation process is performed on the P2 fifth features and the remote sensing label image to obtain P2 sets of predicted global frequency distributions and a set of true global frequency distributions, including: A hierarchical semantic boundary is set for the remote sensing labeled image, and the semantic boundary has a total boundary level of S, where S is an integer greater than or equal to 0; Cut the semantic boundary at boundary level s along its length and width. Next time, obtain There are 3 labeled image blocks; where s is greater than or equal to 0 and less than or equal to S. It is a set of labeled image patches within a semantic boundary of boundary level s, and ; Sure The first local frequency distribution, the The first local frequency distributions are respectively The true local frequency distribution of pixels of each category in each labeled image patch; According to the above A first local frequency distribution is used to determine the set of true global frequency distributions; Repeat the above steps for P2 fifth features to obtain the predicted global frequency distribution of P2 groups.

6. A remote sensing image semantic segmentation model, trained by the method described in any one of claims 1-5, characterized in that, include: An encoder is used to acquire remote sensing images; The remote sensing image is subjected to L groups of first processing to obtain L downsampling first processing results with progressively decreasing resolution; wherein, the last first processing result is the first feature. Each group of first processing in the L groups includes at least one convolutional processing, and the total number of convolutional processing in all groups of first processing is M. The stride of the convolution kernel used in M1 of the M convolutional processing is greater than 1, where L is a positive integer, M is a positive integer, and M1 is a positive integer greater than 1 and less than M; wherein, the last downsampling processing result is the first feature. The decoder is used to perform L1 upsampling processes on the first feature to obtain L1 upsampling results with progressively increasing resolution, where L1 is a positive integer; the last upsampling result is the second feature; and the second feature is used to classify each pixel to determine the class probability of each pixel in the second feature. A multi-scale spatial detail (MSD) enhancement unit is used to perform feature selection processing on P1 third features to obtain P1 fourth features; wherein, the P1 third features include at least one of L first processing results and L1 upsampling processing results; a remote sensing label image corresponding to the remote sensing image is obtained, wherein the labels in the remote sensing label image are used to characterize the category of each pixel therein; based on the labels, the spatial detail features of the remote sensing label image are extracted to obtain a sixth feature; The hierarchical semantic enhancement (HSE) unit is used to acquire remote sensing label images corresponding to remote sensing images; hierarchical semantic boundary segmentation processing is performed on P2 fifth features and the remote sensing label images respectively to obtain P2 sets of predicted global frequency distributions and a set of true global frequency distributions, where P2 is a positive integer, and the P2 fifth features include at least one of L first processing results and L1 upsampling processing results.

7. The remote sensing image semantic segmentation model according to claim 6, characterized in that, The Multi-Scale Spatial Detail (MSD) Enhancement Unit is specifically used to select features from at least one channel of each third feature in the same proportion to obtain P1 fifth features. The P1 fifth features are subjected to a first convolution process to obtain P1 fourth features.

8. The remote sensing image semantic segmentation model according to claim 6, characterized in that, The hierarchical semantic enhancement (HSE) unit is specifically used to set hierarchical semantic boundaries for the remote sensing label image. The semantic boundaries have a total boundary level of S, where S is an integer greater than or equal to 0. The semantic boundaries at boundary level s are then cut along their length and width. Next time, obtain There are 3 labeled image blocks; where s is greater than or equal to 0 and less than or equal to S. It is a set of labeled image patches within a semantic boundary of boundary level s, and ; Sure The first local frequency distribution, the The first local frequency distributions are respectively The true local frequency distribution of pixels of each category in each label image block; according to the A first local frequency distribution is used to determine the set of true global frequency distributions; the aforementioned steps are repeated for P2 fifth features to obtain P2 sets of predicted global frequency distributions.

9. An electronic device, characterized in that, It includes at least one processor for executing a program stored in a memory, which, when executed, implements the method as described in any one of claims 1-5.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images