Remote sensing image building segmentation method and device based on multilayer feature fusion and foreground perception enhancement

Abstract

The invention discloses a remote sensing image building segmentation method and device based on multilayer feature fusion and foreground perception enhancement. The method comprises the following steps: inputting a to-be-segmented image into a backbone network to extract feature information of a building; carrying out feature fusion on the extracted features of different levels; establishing a relation between foreground pixels and a scene through foreground perception enhancement, thereby improving a correlation degree between foreground classes in the fusion feature map; the attention degree of the building foreground in the feature map is improved by using a double-path attention mechanism; and separating a building in the remote sensing image according to the feature map after multilayer feature fusion, foreground perception enhancement and double-path attention processing, and outputting a final segmented image. The method can effectively solve the problems that the small target building in the remote sensing image is fuzzy in feature and difficult to segment, the intra-class difference is large, and misinformation is likely to be generated, and provides an effective solution for remote sensing image building segmentation under the complex background.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a remote sensing image building segmentation method and device based on multi-layer feature fusion and foreground perception enhancement. Background Art

[0002] Remote sensing image building segmentation aims to accurately segment building targets in remote sensing images with complex backgrounds. With the development of remote sensing technology, high-resolution remote sensing images are playing an increasingly important role in urban construction, environmental monitoring and other fields. As an important type of land object in urban remote sensing images, accurate extraction and analysis of buildings are of great significance for urban planning, geographic information system updates, disaster assessment and other aspects. Existing image semantic segmentation methods, such as threshold-based segmentation, region-based segmentation and edge-based segmentation, can provide reasonable segmentation results in some cases, but when processing remote sensing images with complex backgrounds and numerous targets, the performance of these methods is often severely reduced.

[0003] In recent years, the mainstream remote sensing image building segmentation methods are mainly divided into methods based on Convolutional Neural Network (CNN), methods based on Transformer, and methods based on CNN-Transformer hybrid structures according to the network structure adopted. Among them, the latest remote sensing image building segmentation methods mainly adopt CNN-Transformer hybrid structures. These methods use the complementary advantages of self-attention and convolutional attention to enable the model to capture more key feature information of buildings. Although the above methods have improved the performance of remote sensing image building segmentation to a certain extent, they cannot effectively solve the problem of false alarms caused by the fuzzy and difficult to extract features of small target buildings in remote sensing images and the large intra-class differences, and have certain limitations.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] (1) In remote sensing images, there are a large number of small target buildings that only occupy a few pixels in the image. In addition, since remote sensing images often have complex backgrounds, small target buildings are easily confused with the background and are difficult to segment accurately.

[0006] (2) Compared with conventional semantic segmentation tasks, high-resolution remote sensing image building semantic segmentation faces greater intra-class differences and is therefore more prone to false positives.

[0007] (3) Current remote sensing image building semantic segmentation methods find it difficult to simultaneously obtain local detail information and global context information.

[0008] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0009] The purpose of the present invention is to provide a remote sensing image building segmentation method and device based on multi-layer feature fusion and foreground perception enhancement, so as to achieve accurate semantic segmentation of remote sensing image building targets under complex backgrounds.

[0010] In order to achieve the above objectives, the present invention provides a remote sensing image building segmentation method and device based on multi-layer feature fusion and foreground perception enhancement, comprising the following steps:

[0011] Step S1: Input the image to be segmented into the backbone network to extract the feature information of the building;

[0012] Step S2: Fusing the extracted features at different levels;

[0013] Step S3: Establishing the connection between foreground pixels and scenes through foreground perception enhancement, thereby improving the correlation between foreground classes in the fusion feature map;

[0014] Step S4: Use the dual-path attention mechanism to increase the attention to the building foreground in the feature map;

[0015] Step S5: Separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image.

[0016] Furthermore, the process of feature fusion in step S2 is as follows: first, the features C2, C3, and C4 of the 2nd, 3rd, and 4th layers of the backbone network are upsampled to the same size as the first layer feature C1, and then the upsampled C2, C3, and C4 are processed by the smoothing function ω(·) to obtain smooth features F of the same size. i , which is calculated as follows:

[0017]

[0018] Among them, Up(·) represents the upsampling operation, and the smoothing function ω(·) is implemented by 3×3 convolution, batch normalization and ReLU activation function; then the smoothed feature F i After splicing, the spliced ​​feature map is subjected to dimension reduction and feature fusion by f(·) to output the feature map M after multi-layer feature fusion. The calculation method is as follows:

[0019] M=f(Concatenate(F1,F2,F3,F4))

[0020] Among them, Concatenate(·) means concatenating F according to the channel dimension, and f(·) is implemented by two consecutive sets of 3×3 convolutions, batch normalization, and ReLU activation functions.

[0021] Furthermore, the process of foreground perception enhancement in step S3 is as follows: first, explicitly model the relationship between the object and the scene, then use the potential scene embedding to associate the foreground with the relevant context, and finally use the relationship mapping to re-weight the input feature map; that is, by i Recode and then use the relational mapping r i Reweight it to obtain the feature map e after foreground relationship enhancement i Specifically, first, the feature map d i After the scale-aware projection function ψ(·), we get d i ', which is calculated as follows:

[0022] d i '=ψ(d i )

[0023] The scale-aware projection function ψ(·) is implemented by 1×1 convolution, batch normalization, and ReLU activation function; then, D4 is used to calculate a one-dimensional scene embedding vector u to interact with the foreground through η(·), which is calculated as follows:

[0024] u=η(D4)

[0025] where η(·) is a projection function representing the geospatial scene, implemented by a learnable 1×1 convolutional layer, followed by embedding the scene into vectors u and d i 'pass Calculate r i , which is calculated as follows:

[0026]

[0027] in, Represents a similar estimation function, which is implemented by dot product in the present invention; Finally, the foreground and background feature map e of the relationship enhancement i The calculation method of is as follows:

[0028]

[0029] Among them, k wi (·) is the input feature map d i The learnable parameter w iThe encoder is designed to introduce an additional nonlinear unit to avoid feature degradation. The present invention adopts 1×1 convolution, batch normalization and ReLU activation function to achieve it.

[0030] Furthermore, the dual-path attention described in step S4 includes two parallel paths of global attention and local attention; specifically, for the global attention branch, the input feature map is first subjected to the self-attention after the spatial compression operation, and then downsampled by 4 times using 4×4 convolution to establish a coarse-grained global context dependency; for the local attention branch, multi-scale local features are first extracted through 3×3 regular convolution, 1×3 and 3×1, 1×5 and 5×1, 1×7 and 7×1 three groups of strip convolutions, and then they are added and fused and dot-multiplied with the original input to obtain local spatial details; finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion; the overall calculation process of the dual-path attention is as follows:

[0031]

[0032] Among them, SR represents the spatial compression operation, FFN represents the feedforward network, and A GPA represents the output of the global attention branch, A LPA represents the output of the local attention branch, and A represents the final output of the dual-path attention.

[0033] Based on the same inventive concept, an embodiment of the present invention further provides a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which is composed of five modules, including:

[0034] A feature extraction module is used to perform preliminary feature extraction on the input image through the backbone network;

[0035] A multi-layer feature fusion module is used to fuse the features of different levels extracted by the feature extraction module;

[0036] The foreground-aware refinement module is used to establish the connection between foreground pixels and the scene through foreground-aware enhancement, thereby improving the correlation between foreground classes in the fusion feature map;

[0037] Dual-path attention module, which is used to improve the attention to the building foreground in the feature map using the dual-path attention mechanism;

[0038] The image segmentation module is used to separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image.

[0039] In a possible implementation, the multi-layer feature fusion module upsamples deep features at different levels and fuses them with shallow features containing more building detail information and small target information.

[0040] In one possible implementation, the process of the foreground-aware refinement module is: first, explicitly model the object-scene relationship, then use the latent scene embedding to associate the foreground with the relevant context, and finally use the relationship mapping to re-weight the input feature map.

[0041] In a possible implementation, the dual-path attention module includes two parallel paths, global attention and local attention. For the global attention branch, the input feature map is first subjected to a spatial compression operation in the self-attention, and then downsampled by 4 times using a 4×4 convolution to establish a coarse-grained global context dependency. For the local attention branch, multi-scale local features are first extracted through 3×3 regular convolutions, 1×3 and 3×1, 1×5 and 5×1, and 1×7 and 7×1 strip convolutions. These are then added and fused and then dot-multiplied with the original input to obtain local spatial details. Finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion.

[0042] The above five modules are encapsulated and deployed to the server platform to obtain a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which is used to achieve the segmentation of building targets in remote sensing images under complex backgrounds.

[0043] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0044] (1) The present invention enables the network to more effectively utilize shallow feature information containing more small target feature information by fusing shallow features at different levels, thereby improving the segmentation effect of small target buildings;

[0045] (2) The foreground perception enhancement proposed in the present invention enhances the connection between foreground pixels by modeling the target scene, effectively reducing the false alarm problem caused by excessive intra-class differences;

[0046] (3) The dual-path attention proposed in the present invention enables the model to simultaneously obtain local information and establish global contextual dependencies through the complementarity between local attention and self-attention, effectively improving the segmentation performance of the model;

[0047] (4) The present invention proposes a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which can effectively solve the problems of low segmentation accuracy and false alarm of small target buildings in remote sensing images, thereby providing efficient and accurate segmentation of building targets in remote sensing images under complex backgrounds.

[0048] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 It is a flow chart of a remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement in an embodiment of the present invention;

[0051] Figure 2 A network structure diagram of a remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement in an embodiment of the present invention;

[0052] Figure 3 A network structure diagram of multi-layer feature fusion in an embodiment of the present invention;

[0053] Figure 4 A network structure diagram of foreground perception refinement in an embodiment of the present invention;

[0054] Figure 5 A network structure diagram of dual-path attention in an embodiment of the present invention;

[0055] Figure 6 1 is a comparison chart of the segmentation effects of the method of the present invention and other mainstream remote sensing image building segmentation methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise expressly stated, throughout the specification and claims, the term "including" or its variations such as "comprising" or "including" will be understood to include the stated elements or components, without excluding other elements or other components.

[0057] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0058] In addition, in order to better illustrate the present invention, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, and elements well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0059] According to an embodiment of the present invention, a remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement is provided. The flowchart of the method is as follows: Figure 1 As shown, the overall network structure is as follows Figure 2 As shown, the specific steps include:

[0060] Step 101: Input the image to be segmented into the backbone network to extract the feature information of the building;

[0061] In this embodiment, the backbone network uses ConvNeXt to extract initial feature information of the building.

[0062] Step 102: Fusing the extracted features at different levels;

[0063] In this embodiment, the main idea of ​​the feature fusion is to upsample the deep features at different levels and fuse them with the shallow features containing more building detail information and small target information, so as to enhance the shallow feature information and enable the network to focus on the important semantic information of small targets more effectively. The network structure is as follows: Figure 3 Specifically, the main process is as follows: First, in order to merge the low-resolution deep features with the high-resolution shallow features, the features C2, C3, and C4 of the 2nd, 3rd, and 4th layers of the backbone network are upsampled to the same size as the first-layer feature C1. Subsequently, the upsampled C2, C3, and C4 are processed by the smoothing function ω(·) to obtain smooth features F of the same size. i , which is calculated as follows:

[0064]

[0065] Among them, Up(·) represents the upsampling operation, and the smoothing function ω(·) is implemented by 3×3 convolution, batch normalization and ReLU activation function; then the smoothed feature F i After splicing, the spliced ​​feature map is subjected to dimension reduction and feature fusion by f(·) to output the feature map M after multi-layer feature fusion. The calculation method is as follows:

[0066] M=f(Concatenate(F1,F2,F3,F4))

[0067] Among them, Concatenate(·) means concatenating F according to the channel dimension, and f(·) is implemented by two consecutive sets of 3×3 convolutions, batch normalization, and ReLU activation functions.

[0068] Step 103: Establishing a connection between foreground pixels and scenes through foreground perception enhancement, thereby improving the correlation between foreground classes in the fusion feature map;

[0069] In this embodiment, the foreground perception enhancement enhances the network's ability to distinguish foreground features by utilizing foreground context information related to the object-scene relationship, thereby improving the segmentation performance of the model. The network structure is as follows: Figure 4 The main idea is to first explicitly model the relationship between objects and scenes, then use potential scene embedding to associate the foreground with the relevant context, and finally use the relationship mapping to re-weight the input feature map to enhance the difference between foreground features and background features, thereby improving the network's ability to discriminate foreground features. i Recode and then use the relational mapping r i Reweight it to obtain the feature map e after foreground relationship enhancement i Specifically, first, the feature map d i After the scale-aware projection function ψ(·), we get d i ', which is calculated as follows:

[0070] d i '=ψ(d i )

[0071] The scale-aware projection function ψ(·) is implemented by 1×1 convolution, batch normalization, and ReLU activation function; then, D4 is used to calculate a one-dimensional scene embedding vector u to interact with the foreground through η(·), which is calculated as follows:

[0072] u=η(D4)

[0073] where η(·) is a projection function representing the geospatial scene, implemented by a learnable 1×1 convolutional layer, followed by embedding the scene into vectors u and d i 'pass Calculate r i , which is calculated as follows:

[0074]

[0075] in, Represents a similar estimation function, which is implemented by dot product in the present invention; Finally, the foreground and background feature map e of the relationship enhancement i The calculation method of is as follows:

[0076]

[0077] Among them, k wi (·) is the input feature map d i The learnable parameter w i The encoder is designed to introduce an additional nonlinear unit to avoid feature degradation. The present invention adopts 1×1 convolution, batch normalization and ReLU activation function to achieve it.

[0078] Step 104: Use a dual-path attention mechanism to increase the attention paid to the foreground of the building in the feature map;

[0079] In this embodiment, the dual-path attention includes two parallel paths: global attention and local attention. The network structure is as follows: Figure 5 As shown in the figure, the main idea is to reduce the resolution of the feature image after the spatial compression operation to further reduce the sequence length, obtain a rough global context dependency, thereby further reducing the computational cost, and then supplement the local detail information through convolutional attention. Specifically, for the global attention branch, the input feature map is first subjected to a spatial compression operation and then downsampled by 4 times using a 4×4 convolution to further reduce the computational cost of self-attention and establish a coarse-grained global context dependency. For the local attention branch, multi-scale local features are first extracted through 3×3 regular convolutions, 1×3 and 3×1, 1×5 and 5×1, 1×7 and 7×1 strip convolutions, and then they are added and fused and then dot-multiplied with the original input to obtain local spatial details, providing detailed information supplement for global attention. Finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion. The overall calculation process of dual-path attention is as follows:

[0080]

[0081] Among them, SR represents the spatial compression operation, FFN represents the feedforward network, and A GPA represents the output of the GPA branch, A LPA represents the output of the LPA branch, and A represents the final output of the dual-path attention.

[0082] Step 105: Separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image;

[0083] In this embodiment, in order to evaluate the comprehensiveness, four indicators widely used in remote sensing image building segmentation, namely, precision, recall, F1 score, and intersection over union (IoU), are used to evaluate the performance of the model in the present invention. Their calculation formulas are as follows:

[0084]

[0085] Among them, TP, FP and FN represent True Positive, False Positive and False Negative respectively.

[0086] The performance comparison results of the proposed method and the current mainstream remote sensing building segmentation methods on three public remote sensing building semantic segmentation datasets, namely Massachusetts, WHU and Inria, are shown in Table 1, where the bold in the table represents the best results, the data with “_” represents the suboptimal results, and the “-” mark indicates that the method has not been subjected to corresponding experiments.

[0087] Table 1

[0088]

[0089] In order to more intuitively demonstrate the segmentation effect of the proposed method on building targets in remote sensing images, the remote sensing building image segmentation method of the same type is randomly selected to be tested on the Massachusetts remote sensing building semantic segmentation dataset and its segmentation effect is compared and visualized. Figure 6 As shown in the figure, the test images are randomly selected, each row in the figure represents the segmentation effect of different methods on the same image, and each column represents the segmentation effect of the same method on different images.

[0090] The embodiment of the present invention provides a remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement, which effectively solves the problem that the background of remote sensing images is complex and the targets are numerous, so that small target buildings are difficult to segment and the intra-class differences are too large, and can accurately segment building targets in remote sensing images.

[0091] The above introduces in detail a remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement. Based on this method, a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement is proposed. The structure and deployment process of the device are introduced in detail below.

[0092] The embodiment of the present invention further provides a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which is composed of five modules, including:

[0093] A feature extraction module is used to input the image to be segmented into the backbone network to extract the feature information of the building;

[0094] A multi-layer feature fusion module is used to fuse the features of different levels extracted by the feature extraction module;

[0095] The foreground-aware refinement module is used to establish the connection between foreground pixels and the scene through foreground-aware enhancement, thereby improving the correlation between foreground classes in the fusion feature map;

[0096] Dual-path attention module, which is used to improve the attention to the building foreground in the feature map using the dual-path attention mechanism;

[0097] The image segmentation module is used to separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image;

[0098] In a possible implementation, the main idea of ​​the multi-layer feature fusion module is to fuse shallow features at different levels, enhance shallow feature information, and enable the network to more effectively focus on the important semantic information of small targets; specifically, its main process is: first, in order to fuse low-resolution deep features with high-resolution shallow features, the features C2, C3, and C4 of the 2nd, 3rd, and 4th layers of the backbone network are upsampled to the same size as the first layer feature C1, and then the upsampled C2, C3, and C4 are processed by the smoothing function ω(·) to obtain smooth features F of the same size. i , which is calculated as follows:

[0099]

[0100] Among them, Up(·) represents the upsampling operation, and the smoothing function ω(·) is implemented by 3×3 convolution, batch normalization and ReLU activation function; then the smoothed feature F i After splicing, the spliced ​​feature map is subjected to dimension reduction and feature fusion by f(·) to output the feature map M after multi-layer feature weighted fusion. The calculation method is as follows:

[0101] M=f(Concatenate(F1,F2,F3,F4))

[0102] Among them, Concatenate(·) means concatenating F according to the channel dimension, and f(·) is implemented by two consecutive sets of 3×3 convolutions, batch normalization, and ReLU activation functions.

[0103] In a possible implementation, the foreground perception enhancement module enhances the network's ability to discriminate foreground features by utilizing foreground context information related to the object-scene relationship, thereby improving the segmentation performance of the model; the main idea is to first explicitly model the object-scene relationship, then use potential scene embedding to associate the foreground with the relevant context, and finally use the relationship mapping to re-weight the input feature map to enhance the difference between foreground features and background features, thereby improving the network's ability to discriminate foreground features, that is, by re-weighting the feature map d i Recode and then use the relational mapping r i Reweight it to obtain the feature map e after foreground relationship enhancement i Specifically, first, the feature map d i After the scale-aware projection function ψ(·), we get d i ', which is calculated as follows:

[0104] d i '=ψ(d i )

[0105] The scale-aware projection function ψ(·) is implemented by 1×1 convolution, batch normalization, and ReLU activation function; then, D4 is used to calculate a one-dimensional scene embedding vector u to interact with the foreground through η(·), which is calculated as follows:

[0106] u=η(D4)

[0107] where η(·) is a projection function representing the geospatial scene, implemented by a learnable 1×1 convolutional layer, followed by embedding the scene into vectors u and d i 'pass Calculate r i , which is calculated as follows:

[0108]

[0109] in, Represents a similar estimation function, which is implemented by dot product in the present invention; Finally, the foreground and background feature map e of the relationship enhancement i The calculation method of is as follows:

[0110]

[0111] Among them, k wi (·) is the input feature map d i The learnable parameter w i The encoder is designed to introduce an additional nonlinear unit to avoid feature degradation. The present invention adopts 1×1 convolution, batch normalization and ReLU activation function to achieve it.

[0112] In a possible implementation, the dual-path attention module includes two parallel paths of global attention and local attention; the main idea is to reduce the resolution of the feature image after the spatial compression operation to further reduce the sequence length, obtain a rough global context dependency, thereby further reducing the computational cost, and then supplement the local detail information through convolutional attention; specifically, for the global attention branch, the input feature map is first subjected to a spatial compression operation and then downsampled by 4 times using a 4×4 convolution to further reduce the computational cost of self-attention and establish a coarse-grained global context dependency; for the local attention branch, multi-scale local features are first extracted through 3×3 conventional convolutions, 1×3 and 3×1, 1×5 and 5×1, 1×7 and 7×1, and three groups of strip convolutions, and then they are added and fused and dot-multiplied with the original input to obtain local spatial details, providing detailed information supplement for the global attention; finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion; the overall calculation process of the dual-path attention is shown in the following formula:

[0113]

[0114] Among them, SR represents the spatial compression operation, FFN represents the feedforward network, and A GPA represents the output of the global attention branch, A LPA represents the output of the local attention branch, and A represents the final output of the dual-path attention.

[0115] The above five modules are encapsulated and deployed to the server platform to obtain a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which is used for the segmentation of building targets in remote sensing images under complex backgrounds.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and according to the above steps, corresponding changes can be made to it in form and detail, further enabling those skilled in the art to use the present invention to implement various embodiments or make various changes and selections. The scope of the present invention is intended to be defined by the claims and their equivalents.

Claims

1. A remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement, characterized in that: The following steps are involved: Step S1: Input the image to be segmented into the backbone network to extract the feature information of the building; Step S2: Fusing the extracted features at different levels; Step S3: Establishing the connection between foreground pixels and scenes through foreground perception enhancement, thereby improving the correlation between foreground classes in the fusion feature map; Step S4: Use the dual-path attention mechanism to increase the attention to the building foreground in the feature map; Step S5: Separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image.

2. The remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement according to claim 1 is characterized in that: The process of feature fusion in step S2 is as follows: first, the features C2, C3, and C4 of the 2nd, 3rd, and 4th layers of the backbone network are upsampled to the same size as the first layer feature C1, and then the upsampled C2, C3, and C4 are processed by the smoothing function ω(·) to obtain smooth features F of the same size. i , which is calculated as follows: Among them, Up(·) represents the upsampling operation, and the smoothing function ω(·) is implemented by 3×3 convolution, batch normalization and ReLU activation function; then the smoothed feature F i After splicing, the spliced ​​feature map is subjected to dimension reduction and feature fusion by f(·) to output the feature map M after multi-layer feature fusion. The calculation method is as follows: M=f(Concatenate(F1,F2,F3,F4)) Among them, Concatenate(·) means concatenating F according to the channel dimension, and f(·) is implemented by two consecutive sets of 3×3 convolutions, batch normalization, and ReLU activation functions.

3. The remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement according to claim 1 is characterized in that: The process of foreground perception enhancement in step S3 is as follows: first, explicitly model the object-scene relationship, then use the potential scene embedding to associate the foreground with the relevant context, and finally use the relationship mapping to reweight the input feature map; that is, by i Recode and then use the relational mapping r i Reweight it to obtain the feature map e after foreground relationship enhancement i Specifically, first, the feature map d i After the scale-aware projection function ψ(·), we get d i ', which is calculated as follows: d i '=ψ(d i ) The scale-aware projection function ψ(·) is implemented by 1×1 convolution, batch normalization, and ReLU activation function; then, D4 is used to calculate a one-dimensional scene embedding vector u to interact with the foreground through η(·), which is calculated as follows: u=η(D4) where η(·) represents the projection function of the geospatial scene, which is implemented by a learnable 1×1 convolutional layer, followed by the scene embedding vectors u and d i 'pass Calculate r i , which is calculated as follows: in, Represents a similar estimation function, which is implemented using dot product; finally, the relationship-enhanced feature map e i The calculation method of is as follows: Among them, k wi (·) is the input feature map d i The learnable parameter w i The encoder is implemented by using 1×1 convolution, batch normalization and ReLU activation function.

4. The remote sensing image building segmentation method based on multi-layer feature fusion and foreground perception enhancement according to claim 1 is characterized in that: The dual-path attention described in step S4 includes two parallel paths: global attention and local attention. Specifically, for the global attention branch, the input feature map is first subjected to self-attention after spatial compression operation, and then downsampled by 4 times using 4×4 convolution to establish a coarse-grained global context dependency. For the local attention branch, multi-scale local features are first extracted through 3×3 regular convolution, 1×3 and 3×1, 1×5 and 5×1, 1×7 and 7×1 strip convolutions, and then they are added and fused and dot-multiplied with the original input to obtain local spatial details; finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion; the overall calculation process of the dual-path attention is as follows: Among them, SR represents the spatial compression operation, FFN represents the feedforward network, and A GPA represents the output of the global attention branch, A LPA represents the output of the local attention branch, and A represents the final output of the dual-path attention.

5. A remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, characterized in that: The device consists of five modules, including: A feature extraction module is used to perform preliminary feature extraction on the input image through the backbone network; A multi-layer feature fusion module is used to fuse the features of different levels extracted by the feature extraction module; The foreground-aware refinement module is used to establish the connection between foreground pixels and the scene through foreground-aware enhancement, thereby improving the correlation between foreground classes in the fusion feature map; Dual-path attention module, which is used to improve the attention to the building foreground in the feature map using the dual-path attention mechanism; The image segmentation module is used to separate the buildings in the remote sensing image according to the feature map after multi-layer feature fusion, foreground perception enhancement and dual-path attention processing, and output the final segmented image; In the multi-layer feature fusion module, deep features at different levels are upsampled and fused with shallow features containing more building detail information and small target information; In the foreground-aware refinement module, the object-scene relationship is first explicitly modeled, then the foreground is associated with the relevant context using latent scene embedding, and finally the input feature map is re-weighted using the relational mapping; The dual-path attention module includes two parallel paths, global attention and local attention. For the global attention branch, the input feature map is first subjected to a spatial compression operation in the self-attention, and then downsampled by 4 times using a 4×4 convolution to establish a coarse-grained global context dependency. For the local attention branch, multi-scale local features are first extracted through 3×3 regular convolutions, 1×3 and 3×1, 1×5 and 5×1, and 1×7 and 7×1 strip convolutions, and then added and fused, and then dot-multiplied with the original input to obtain local spatial details. Finally, the global feature information collected by the global attention and the local feature information obtained by the local attention are added and input into the feedforward network for further information fusion. The above five modules are encapsulated and deployed to the server platform to obtain a remote sensing image building segmentation device based on multi-layer feature fusion and foreground perception enhancement, which is used to achieve the segmentation of remote sensing image buildings.

Images