Remote sensing image semantic segmentation method, system, equipment and medium

Abstract

The invention discloses a remote sensing image semantic segmentation method, system and device and a medium, and the method comprises the following steps: obtaining remote sensing building data, and inputting the data into a pre-trained semantic segmentation network, so as to achieve the precise segmentation of a building in a remote sensing image. The semantic segmentation network includes an encoder module and a decoder module for adaptive context selection based on structure priori guidance. In the training stage, a special remote sensing building data set is utilized to train the network so as to optimize the segmentation performance of the network. According to the method, by introducing structure prior information and an adaptive context selection mechanism, the accuracy and efficiency of building segmentation in the remote sensing image are effectively improved, and the problem that remote sensing buildings are difficult to distinguish when pixel visual evidences are weak or the remote sensing buildings belong to small space structure objects is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image semantic segmentation method, system, equipment and medium. Background Art

[0002] With the rapid progress of remote sensing technology, especially the widespread application of high spatial resolution remote sensing image technology, the ability of earth observation has reached an unprecedented level. With its fine observation details and rich information, high-resolution remote sensing images provide powerful data support for geospatial analysis, environmental monitoring, urban planning, military reconnaissance and other fields. However, how to efficiently and accurately extract building information from these massive and complex image data has become a technical problem that needs to be solved urgently.

[0003] Traditional image processing and segmentation methods, such as threshold-based, edge detection or region growing techniques, are inadequate for high-resolution remote sensing images, mainly because these methods are difficult to effectively handle complex textures, changing shadows, and high overlap and occlusion between buildings in images. Therefore, researchers have gradually turned their attention to deep learning, especially convolutional neural networks (CNNs), a powerful feature extraction and pattern recognition tool.

[0004] Through multi-layer convolution operations, convolutional neural networks can automatically learn multi-level feature representations in images and demonstrate excellent performance in tasks such as image classification, target detection, and semantic segmentation. However, when directly applied to building segmentation in high-resolution remote sensing images, traditional CNN models still face many challenges: first, the texture complexity and spatial structure diversity of buildings in remote sensing images are extremely high, requiring the model to have strong feature representation capabilities; second, the spatial relationships between buildings and between buildings and the background are complex, requiring the model to be able to capture long-distance dependencies and contextual information; finally, interference factors such as lighting changes, shadows, and tree occlusions in remote sensing images further increase the difficulty of segmentation.

[0005] To solve these problems, researchers have proposed a variety of improvement strategies based on deep learning. Among them, dilated convolution increases the receptive field of the convolution kernel, allowing the model to capture a wider range of contextual information; effective use of contextual information, such as through pyramid pooling or feature pyramid networks (FPN) and other structures, helps the model understand image content at different scales; and the introduction of the attention mechanism enables the model to dynamically adjust the degree of attention to different areas of the image, thereby improving the accuracy and efficiency of segmentation.

[0006] Although significant research progress has been made in the field of semantic segmentation of buildings in remote sensing images, and various advanced methods and technologies have emerged, existing solutions still seem to be unable to cope with certain specific challenges. A particularly prominent problem is that when the visual features of building pixels in remote sensing images are fuzzy or weak, that is, when the visual evidence of the pixels is insufficient, existing models often find it difficult to accurately identify and segment these buildings. In addition, for those buildings that are small spatial structural objects, due to their small size, unclear outlines, and limited pixel area occupied in the image, they also pose considerable challenges to existing segmentation models. In these cases, the model may mistakenly confuse these small or visually weak buildings with the background or other objects due to the lack of sufficient contextual information or feature representation capabilities, resulting in inaccurate segmentation results. Therefore, how to further improve the model's recognition and segmentation capabilities for such complex situations remains a key issue that needs to be urgently addressed in the field of semantic segmentation of buildings in remote sensing images. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problem that when the pixel visual evidence in the remote sensing building is weak or the building belongs to a small spatial structure object, the existing model is often difficult to distinguish the building in the remote sensing image, and to provide a remote sensing image semantic segmentation method, system, equipment and medium.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A first aspect of the present invention provides a remote sensing image semantic segmentation method, comprising the following steps:

[0010] Input the remote sensing building image into the trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0011] The training method of the pre-built semantic segmentation network is:

[0012] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0013] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0014] Further, the decoder includes four upsampling blocks and four convolution blocks.

[0015] Furthermore, the four upsampling blocks and the four convolution blocks of the decoder are alternately connected in sequence.

[0016] Furthermore, the convolution block of the encoder includes two convolution layers and a maximum pooling layer; the output ends of the two convolution layers are connected to the input end of the maximum pooling layer; and the output end of the maximum pooling layer is connected to an adaptive context selection module.

[0017] Further, the adaptive context selection module is specifically:

[0018] Based on the U-net encoder-decoder structure semantic segmentation network, the remote sensing building image is processed through the convolution block of the encoder to obtain the first feature map;

[0019] The second feature map is obtained by performing large kernel convolution of different scales on the first feature map and weighting it with the corresponding spatial selection mask;

[0020] The third feature map is obtained by performing strip pooling aggregation in two dimensions, vertically and horizontally, and decoding the first feature map after strip pooling aggregation to the original size using an upsampling layer;

[0021] The second feature map and the third feature map are multiplied element-wise to obtain the final output feature map of the adaptive context selection module.

[0022] Furthermore, the attention feature of the second feature map is described as follows:

[0023]

[0024] in, Represents a dilated convolution with a kernel size of 5×5 and a dilation rate of 1; represents a dilated convolution with a kernel size of 7×7 and a dilation rate of 3; σ() represents a Sigmoid operation, P max represents the maximum pooling, P avg represents average pooling;

[0025] The attention feature description of the third feature map is:

[0026]

[0027] in, H is the height of the input image, W is the width of the input image, represents a pooling operation with a kernel size of 1×W, represents an H×1 pooling operation, and UP represents upsampling.

[0028] Furthermore, the structural prior variable loss function of the pre-constructed semantic segmentation network is:

[0029]

[0030] in: For pixel constraints, the expression is as follows:

[0031]

[0032] is a global constraint, and its expression is as follows:

[0033]

[0034] Among them, λ is a hyperparameter, p i,j is the predicted value of the pixel at position (i, j) in the image, gt i,j is the true label value of the pixel at position (i, j) in the image, a i,j It is the adaptive pixel weight value according to different divided areas, H is the height of the input image, W is the width of the input image, C is the number of channels of the input image, and k is the initial value of the number of channels taken.

[0035] A second aspect of the present invention provides a remote sensing image semantic segmentation system, comprising:

[0036] A semantic segmentation unit, used for inputting the remote sensing building image into a trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0037] The training method of the pre-built semantic segmentation network is:

[0038] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0039] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0040] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote sensing image semantic segmentation method described above is implemented.

[0041] A fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned remote sensing image semantic segmentation method when executing the computer program.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention discloses a remote sensing image semantic segmentation method, which utilizes structural prior knowledge and can accurately capture the spatial structure and detail features of buildings in remote sensing images through an adaptive context selection mechanism. The encoder module effectively extracts multi-scale features in the image and screens and enhances the features in combination with structural prior information; the decoder module gradually restores a high-precision segmentation map of the building based on the output of the encoder. This design enables the model to accurately distinguish between buildings and other objects in the face of challenges such as complex backgrounds and shadow occlusions, achieving high-precision segmentation.

[0044] Furthermore, the encoder part can efficiently extract feature information from remote sensing images through the design of convolution blocks and four adaptive context selection modules. The convolution block is responsible for basic feature extraction, while the adaptive context selection module further enhances the model's ability to understand and select image context information, helping to more accurately capture complex semantic information in remote sensing images. The decoder part achieves accurate upsampling and refinement of the feature map extracted by the encoder through the design of four upsampling blocks and convolution blocks, thereby obtaining high-precision semantic segmentation results. The upsampling block can gradually restore the resolution of the feature map, while the convolution block further refines the features to ensure the accuracy of the segmentation boundary and the consistency of semantics.

[0045] Furthermore, the convolutional block of the encoder can effectively extract low-level and mid-level features in remote sensing images by designing two convolutional layers. The convolutional layer extracts features from the image through convolution operations and generates feature maps. This multi-level convolutional design enables the model to capture more details and texture information in the image. The maximum pooling layer plays a role in feature dimensionality reduction and abstraction in the convolutional block. It reduces the resolution of the feature map by selecting the maximum value in each local area while retaining important feature information. This helps to reduce the amount of computation and allows the model to focus more on the significant features in the image. The output of the maximum pooling layer is connected to an adaptive context selection module, which enables the model to adaptively select important context information based on the characteristics of the input image. This design enhances the model's ability to understand complex scenes in remote sensing images and helps improve the accuracy of semantic segmentation.

[0046] Furthermore, each adaptive context selection module contains two large kernel convolutions, which can capture context information in a larger range, thereby enhancing the model's perception of complex scenes in remote sensing images. Large kernel convolutions expand the receptive field, allowing the model to consider more surrounding pixels when processing images, which helps to improve the accuracy of semantic segmentation. The two strip pooling layers in the module work in parallel, providing the model with a variety of feature extraction methods. Strip pooling can pool feature maps along a specific direction, which helps to capture long strip structures or linear features in the image. The spatial selection mechanism, as the aggregation point of the output of large kernel convolution and strip pooling, can make intelligent selections based on the importance of the input features. This mechanism ensures that only the most valuable features are passed to the next layer, thereby improving the overall performance and efficiency of the model.

[0047] Furthermore, by alternately connecting four upsampling blocks and convolution blocks, the decoder can achieve a progressive feature recovery process. Each upsampling block is responsible for gradually increasing the resolution of the feature map, while the convolution block that follows it further refines and optimizes the upsampled feature map. This progressive recovery method helps the model gradually restore the detailed information of the remote sensing image during the decoding process, while reducing the image blur and distortion caused by direct enlargement.

[0048] Furthermore, the adaptive context selection module uses a carefully designed first branch and a second branch. The first branch effectively constructs large convolution kernels of different scales by adopting two-scale large kernel convolutions and decomposing the large kernel into a deep convolution sequence. This method not only reduces the computational complexity, but also enables the model to capture contextual information of different scales in remote sensing images. By fusing the feature maps corresponding to large convolution kernels of different scales, the model can obtain a richer multi-scale receptive field, thereby more comprehensively understanding the complex scenes in the image. Furthermore, the fused features are processed by the average pooling layer and the maximum pooling layer, and spliced ​​in the channel dimension to extract key spatial relationship information and enhance the model's ability to understand spatial features. Finally, by weighting with the spatial selection mask, a large kernel attention feature with strong attention is obtained, which improves the model's attention to important areas. The second branch aggregates the input features through two strip pooling operations in the vertical and horizontal directions, so that the dependencies between longer-distance feature descriptors can be captured. This strip pooling method is particularly suitable for objects with long strips or linear structures such as large buildings in remote sensing images, which helps the model to more accurately identify the boundaries and shapes of these objects. The features are decoded to the original size through the upsampling layer, the spatial consistency of the feature map is maintained, and it is effectively fused with the features of the first branch. This fusion method enables the model to comprehensively utilize multi-scale contextual information and long-distance feature dependencies, further improving the accuracy and robustness of semantic segmentation; the adaptive context selection module achieves a deep fusion of the two feature information by multiplying the large core attention feature with the pooled attention feature element by element. This fusion method not only retains the advantages of each feature, but also improves the model's adaptive context selection ability for complex scenes through mutual complementation and enhancement. The final output feature map more accurately reflects the semantic information in the remote sensing image, providing strong support for subsequent semantic segmentation tasks.

[0049] In summary, the purpose of the present invention is to provide a remote sensing image semantic segmentation method based on adaptive context selection guided by structural prior, wherein the network mentioned above first constructs an adaptive context large kernel selection module based on large kernel convolution and strip pooling, and embeds it into four different encoding stages of the U-net network. The proposed module uses large kernel convolution to superimpose selective adaptive receptive fields on different building scales to effectively capture the rich features of multi-scale buildings. At the same time, the strip pooling module is used to capture the long-distance spatial relationship of pixels contained in the remote sensing image, and the closely related contextual information of the building segmentation is effectively obtained. Then, the three key structural prior knowledge of the building's points, lines, and surfaces are used to establish the regularized prior knowledge constraints for the segmentation of remote sensing images, and a structural prior-guided variable loss function is constructed to guide the network to learn more building detail features, thereby achieving high-precision feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of a remote sensing image semantic segmentation network structure based on adaptive context selection guided by structure priors according to the present invention;

[0052] Figure 2 This is a structural diagram of the adaptive context selection module of the present invention;

[0053] Figure 3 A structural diagram of a variable loss function guided by a structural prior of the present invention;

[0054] Figure 4 This is a schematic diagram of the effect of the structure prior guiding the variable loss function of the present invention;

[0055] Figure 5 This is a segmentation effect diagram of the remote sensing image semantic segmentation method of the present invention;

[0056] Figure 6 It is a schematic diagram of the process of the present invention;

[0057] Figure 7 A schematic diagram of a computer device provided by an embodiment of the present invention;

[0058] Figure 8 The present invention is a block diagram of an electronic device provided according to an embodiment.

[0059] Among them, 60-computer device, 61-processor, 62-memory, 63-computer program, 600-electronic device, 610-processing unit, 620-storage unit, 6201-random access storage unit, 6202-cache storage unit, 6203-read-only storage unit, 6204-program / utility, 6205-program module, 630-bus, 640-display unit, 650-I / O interface, 660-network adapter, 700-external device. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0062] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0063] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0064] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0065] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0066] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0068] The present invention proposes a remote sensing image semantic segmentation method, comprising the following steps:

[0069] Input the remote sensing building image into the trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0070] The training method of the pre-built semantic segmentation network is:

[0071] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0072] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0073] See also Figure 6 The present invention proposes a remote sensing image semantic segmentation method, comprising the following steps:

[0074] S1. Construct a remote sensing building dataset.

[0075] S101. Collect high-resolution remote sensing building image data;

[0076] This paper uses two public remote sensing building image datasets, WHU Building dataset and Inria dataset, to carry out experiments.

[0077] S102, performing data enhancement on the collected remote sensing building images by random cropping, random horizontal and vertical flipping and other methods, and dividing the collected remote sensing building images into a training data set and a test data set according to a certain ratio.

[0078] S2. Build a semantic segmentation network.

[0079] The semantic segmentation network of the present invention constructs an adaptive context selection module (ACS) on the basis of the U-net encoding and decoding structure semantic segmentation network; based on the adaptive context selection module (ACS), a remote sensing image semantic segmentation network (SGACS-Net) based on adaptive context selection guided by structure prior is constructed. Figure 1 shown.

[0080] The semantic segmentation network (SGACS-Net) of the present invention comprises: an encoder module of SGACS-Net and a decoder module of SGACS-Net, wherein:

[0081] The encoder module of SGACS-Net includes a convolution block and an ACS module. The convolution block includes two convolution layers with a convolution kernel of 3×3. The two convolution layers are connected to a maximum pooling layer, and the maximum pooling layer is connected to the ACS module.

[0082] The ACS module consists of large kernel convolution, strip pooling and spatial selection mechanism, which is divided into two parallel branches as follows: Figure 2 As shown. The first branch includes large kernel convolutions of two scales, 5×5 and 7×7. The large kernel convolution constructs a larger convolution kernel by decomposing the large kernel into a series of deep convolutions to obtain large convolution kernels of different scales, thereby capturing contextual information of different scales in remote sensing images; then, the feature maps obtained by large kernel convolutions of different scales are processed by average pooling layer and maximum pooling layer respectively to extract and emphasize important feature information; then, the pooled feature maps are spliced ​​in the channel dimension to form a fused multi-scale feature representation; for the features of the decomposed large kernel sequence, the corresponding spatial selection mask is used for weighted processing. The spatial selection mask is automatically learned according to the spatial distribution and importance of the feature map, and is used to enhance the feature representation of the key area, suppress unimportant background noise, and thus obtain attention features. The second branch performs strip area segmentation on the feature map from the horizontal and vertical coordinate dimensions, improves the network's ability to capture the long-distance dependency relationship of large buildings in the feature map, and mines the correlation of long-distance feature descriptors in the horizontal and vertical coordinate dimensions.

[0083] The feature extraction process of the ACS module is as follows:

[0084] Let X = {x1, x2, …, x n} represents a given feature map, Y = {y1,y2,…,y n} represents the final output feature of the ACS module. After the input enters the ACS module, it is divided into two branches, one branch LK is to select a larger kernel, and the other branch SP is to apply strip pooling.

[0085] In the LK branch, feature maps are first obtained from large convolution kernels of different scales. and The receptive field sizes of different scales are obtained to enhance the network's ability to focus on the compact spatial context area of ​​the target; then, the spatial relationship is effectively extracted through channel-based average and maximum pooling; finally, the features of the decomposed large kernel sequence are weighted by their corresponding spatial selection masks to obtain the attention feature S. The specific process is shown in the following formula:

[0086]

[0087] in, represents a dilated convolution with a kernel size of 5×5 and a dilation rate of 1. represents a dilated convolution with a kernel size of 7×7 and a dilation rate of 3. σ(·) represents a Sigmoid operation, P max represents the maximum pooling, P avg Represents the average pooling operation.

[0088] In the SP branch, the feature map is first input and Through strip pooling aggregation in both vertical and horizontal dimensions, the dependency relationship of feature descriptors with longer distances is obtained. Then, upsampling is used to decode it to the original feature map size to obtain the attention G. The specific process is shown in the following formula:

[0089]

[0090] in, and They represent pooling operations with kernel sizes of 1×W and H×1 respectively, and UP stands for upsampling.

[0091] The final output Y of the ACS module is the element-by-element product of the input feature X and (S+G). The specific calculation process is as follows:

[0092] Y=(S+G)·X

[0093] The decoder module of the SGACS-Net includes an upsampling block and a convolution block. The upsampling module completes the progressive spatial resolution restoration of the feature map. In order to eliminate the checkerboard effect caused by deconvolution, each upsampling block contains a bilinear upsampling layer and two convolution layers with a convolution kernel size of 3×3 and a padding of 1. At the end of the encoding, there is a convolution block to achieve the segmentation task of the remote sensing image output feature map.

[0094] S3. Train SGACS-Net. The specific steps include:

[0095] The established high-resolution remote sensing building training dataset is input into the network, and the forward propagation algorithm is used to calculate the loss; the network is trained using a structural prior to guide the variable loss function.

[0096] The structural prior guides the variable loss function structure as follows Figure 3 shown.

[0097] Structural priors guide the variable loss function The definition is as follows:

[0098]

[0099] Among them, λ is a hyperparameter, and its default value is 2. i,j and gt i,j is the predicted value and label true value of the pixel at position (i, j) in the image, α i,j It is the adaptive pixel weight value according to different divided areas, and its calculation process can be defined as:

[0100]

[0101] Among them, s∈[e,b,c,bk] corresponds to the label of the region to which the pixel (i,j) belongs. Represents the number of pixels in the region. Each pixel will be assigned a weight Pixels that are more difficult to segment and account for a smaller proportion correspond to larger Simple pixels will be assigned smaller It can be regarded as an indicator of pixel importance, which is calculated based on the ratio between the total number of target pixels and its global total number of pixels.

[0102] The schematic diagram of the effect of structural prior guiding variable loss function is shown in the attached figure. Figure 4 shown. Figure 4 (a) represents the original image of remote sensing buildings. Figure 4 (b) is the label, Figure 4 (c) is the structure prior guided variable loss weight map, Figure 4 (d) is the superposition effect of the consistency loss weight map and the original remote sensing building image. As can be seen from the figure, compared with BCE, More attention is paid to hard-to-distinguish pixels. In addition, the local structural information of the image has been encoded into This helps the network focus on a larger receptive field rather than a single pixel, avoiding the problem that pixels with weak visual features cannot be recognized.

[0103] In order to further make the network focus on the global structure, a weighted IoU loss is introduced into the loss function.

[0104]

[0105] The IoU loss aims to optimize the global structure instead of focusing on individual pixels, and is therefore not affected by imbalanced distribution. However, the IoU loss still treats all pixels equally and ignores the differences between pixels. In contrast, the wIoU loss used in this paper gives higher weights to hard-to-distinguish pixels to emphasize their importance. The structural prior-guided variable loss integrates local structural information, generates different weights for each pixel, and introduces pixel constraints. and global constraints This can better guide network learning and produce clear details. The structural prior guides the variable loss as shown below:

[0106]

[0107] S4. Input the remote sensing image to be extracted into the trained building extraction network to obtain the building extraction result.

[0108] The present invention also provides a remote sensing image semantic segmentation system, comprising:

[0109] A semantic segmentation unit, used for inputting the remote sensing building image into a trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0110] The training method of the pre-built semantic segmentation network is:

[0111] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0112] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0113] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the substation maintenance and replacement method considering the time value, including:

[0114] Input the remote sensing building image into the trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0115] The training method of the pre-built semantic segmentation network is:

[0116] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0117] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0118] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0119] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0120] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0121] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the substation maintenance and replacement method considering time value in the above embodiment; the processor may load and execute the following steps:

[0122] Input the remote sensing building image into the trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image;

[0123] The training method of the pre-built semantic segmentation network is:

[0124] Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network;

[0125] Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

[0126] See also Figure 7 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the substation maintenance and replacement method considering the time value in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the substation maintenance and replacement system considering the time value in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0127] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 8 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0128] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0129] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0130] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0131] Any reference to memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0132] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0133] See also Figure 8 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0134] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0135] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0136] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0137] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0138] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0139] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0140] See also Figure 5 , Figure 5 The segmentation effect diagram is shown in red, indicating false detection and blue indicating omission. Figure 5 As shown in (a), the SGACS-Net network proposed in the present invention uses a structural prior to guide the variable loss to give high-level structural prior constraints to the edge error-prone areas to enhance the network's recognition ability for weak visual feature pixels; for the building recognition ambiguity problem in the upper left corner area of ​​a large building, as shown in Figure 5 As shown in (b), the SGACS-Net network gives greater weight to the corner areas with a smaller number of pixels through the constraints of high-level structural prior knowledge, and combines the adaptive context selection module to model long-distance dependencies through a larger receptive field, so that large buildings can maintain high integrity and have fewer omissions; for the segmentation effect of long strip buildings, the edge area segmentation is blurred and the edge extends outward, as shown in the segmentation effect map. Figure 5 As shown in (c), the present invention obtains more context information through the adaptive context selection module, captures richer edge structure information of the building, makes its edge segmentation more regular, and has less false detection; for the problem of ignoring small target building groups, such as Figure 5 As shown in (d), the present invention accurately locates the spatial information of small target buildings through the spatial selection mechanism in the adaptive context selection module, so that fewer small targets are missed in the segmentation effect map.

[0141] In summary, the present invention provides a remote sensing image semantic segmentation method, system, device and medium. The proposed SGACS-Net can dynamically adjust the receptive field to adapt to changes in building structures when facing complex environments, reduce false detections and omissions, and has a good segmentation effect.

[0142] The SGACS-Net of the present invention effectively solves the problem of difficulty in distinguishing when the visual evidence of pixels in remotely sensed buildings is weak or they belong to small spatial structure objects, as follows:

[0143] First, the adaptive context selection module ACS is introduced. This module uses two spatial dimension encoding methods, LK large kernel convolution and SP strip pooling, to adaptively adjust the size of the receptive field, thereby obtaining rich contextual prior knowledge and enhancing the network's ability to represent building features in complex scenes.

[0144] Secondly, a structural prior-guided variable loss is proposed. This loss uses three key shape feature factors, namely points, lines, and surfaces, to divide building areas and construct regularized structural prior constraints, thereby further improving the precision and accuracy of image segmentation.

[0145] The present invention constructs an adaptive context selection module based on large kernel convolution and strip pooling, and embeds it into four different encoding stages of the U-net network. The adaptive context selection module uses large kernel convolution to superimpose selective adaptation on different building scales, effectively capturing the rich features of multi-scale buildings. At the same time, the strip pooling module is used to capture the long-distance spatial relationship of pixels contained in remote sensing images, and effectively obtain the closely related context information of building segmentation. Then, the three key structural prior knowledge of points, lines and surfaces of buildings are used to establish the regularized prior knowledge constraints of remote sensing image building segmentation, and the structural prior guided variable loss function is constructed to guide the network to learn more building detail features, thereby achieving high-precision feature extraction.

[0146] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image semantic segmentation method, characterized in that: The following steps are involved: Input the remote sensing building image into the trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image; The training method of the pre-built semantic segmentation network is: Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network; Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

2. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The decoder includes four upsampling blocks and four convolution blocks.

3. The remote sensing image semantic segmentation method according to claim 2, characterized in that: The four upsampling blocks and the four convolution blocks of the decoder are alternately connected in sequence.

4. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The convolution block of the encoder includes two convolution layers and a maximum pooling layer; the output ends of the two convolution layers are connected to the input end of the maximum pooling layer; and the output end of the maximum pooling layer is connected to an adaptive context selection module.

5. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The adaptive context selection module is specifically: Based on the U-net encoder-decoder structure semantic segmentation network, the remote sensing building image is processed through the convolution block of the encoder to obtain the first feature map; The second feature map is obtained by performing large kernel convolution of different scales on the first feature map and weighting it with the corresponding spatial selection mask; The third feature map is obtained by performing strip pooling aggregation in two dimensions, vertically and horizontally, and decoding the first feature map after strip pooling aggregation to the original size using an upsampling layer; The second feature map and the third feature map are multiplied element-wise to obtain the final output feature map of the adaptive context selection module.

6. A remote sensing image semantic segmentation method according to claim 5, characterized in that: The attention feature description of the second feature map is: in, Represents a dilated convolution with a kernel size of 5×5 and a dilation rate of 1; represents a dilated convolution with a kernel size of 7×7 and a dilation rate of 3; σ() represents a Sigmoid operation, P max represents the maximum pooling, P avg represents average pooling; The attention feature description of the third feature map is: in, H is the height of the input image, W is the width of the input image, represents a pooling operation with a kernel size of 1×W, represents an H×1 pooling operation, and UP represents upsampling.

7. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The structural prior variable loss function of the pre-built semantic segmentation network is: in: For pixel constraints, the expression is as follows: is a global constraint, and its expression is as follows: Among them, λ is a hyperparameter, p i,j is the predicted value of the pixel at position (i, j) in the image, gt i,j is the true label value of the pixel at position (i, j) in the image, a i,j It is the adaptive pixel weight value according to different divided areas, H is the height of the input image, W is the width of the input image, C is the number of channels of the input image, and k is the initial value of the number of channels taken.

8. A remote sensing image semantic segmentation system, characterized in that: include: A semantic segmentation unit, used for inputting the remote sensing building image into a trained pre-built semantic segmentation network to segment the buildings in the remote sensing building image; The training method of the pre-built semantic segmentation network is: Acquire a remote sensing building training data set; train a pre-built semantic segmentation network using the acquired remote sensing building training data set to obtain a trained semantic segmentation network; Among them, the semantic segmentation network includes an encoder and a decoder; the encoder includes a convolution block and four adaptive context selection modules; the adaptive context selection module includes two parallel large-kernel convolutions, two parallel strip pooling and a spatial selection mechanism; the large-kernel convolution is connected to the strip pooling in parallel and then connected to the spatial selection mechanism in series.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote sensing image semantic segmentation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the remote sensing image semantic segmentation method according to any one of claims 1 to 7 when executing the computer program.

Images