Remote sensing image building segmentation method and related equipment based on neural network model

Abstract

The embodiment of the present invention provides a remote sensing image building segmentation method based on a neural network model and related equipment, which relates to the technical field of remote sensing image segmentation. The method includes: obtaining a target remote sensing image to be processed; constructing a remote sensing image segmentation model; obtaining N global feature maps of different scales through a multi-scale Transformer module; obtaining N local feature maps of different scales through a CNN downsampling module; merging N global feature maps and N local feature maps into N feature fusion maps; merging the target feature fusion map with N-1 feature fusion maps in turn to obtain an overall feature map, and obtaining a pixel-level segmentation result of the target remote sensing image based on the overall feature map. Through the present invention, the problem of low segmentation accuracy of remote sensing images is solved, thereby achieving the effect of improving the segmentation accuracy of remote sensing images.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of remote sensing image segmentation, and in particular, to a remote sensing image building segmentation method based on a neural network model and related equipment. Background Art

[0002] Building extraction based on fine-resolution remote sensing images, i.e. identifying building and non-building pixels in images, plays a vital role in a wide range of application scenarios such as urban planning, demography, economic assessment, disaster management, etc. Traditional building extraction methods usually extract manual features (such as spectral, spatial, texture, etc.) and apply traditional convolutional neural networks (CNNs) to identify buildings.

[0003] Convolutional neural networks do not use a fully connected approach between the upper and lower layers, but instead use a biological receptive field mechanism. It is a feedforward neural network with a deep structure. Convolutional layers and pooling layers are commonly used in convolutional neural networks. Local connections replace full connections. Convolution operations are performed before each layer, and each convolution layer is connected to a downsampling layer to reduce the number of parameters accordingly. Convolutional neural networks are best at image processing and are mainly used for image recognition and classification tasks. They can effectively reduce the dimension of images and retain image features, and are robust to displacement, scaling or other forms of transformation in two-dimensional images. However, due to the inevitable limitations of the receptive field in the convolution operation and the inherent inductive bias in the convolution architecture, the convolution algorithm cannot establish global context dependencies in the image, thereby affecting the segmentation accuracy of remote sensing images. Summary of the invention

[0004] The embodiment of the present invention provides a remote sensing image building segmentation method based on a neural network model and related equipment, so as to at least solve the problem of low remote sensing image segmentation accuracy in the related art.

[0005] According to one embodiment of the present invention, a remote sensing image building segmentation method based on a neural network model is provided, comprising:

[0006] Acquire the target remote sensing image to be processed;

[0007] Combine the multi-scale Transformer module and the CNN downsampling module to build a remote sensing image segmentation model;

[0008] Inputting the target remote sensing image into the remote sensing image segmentation model, processing the target remote sensing image through N global feature extraction layers of different scales in the multi-scale Transformer module, and obtaining N global feature maps of different scales;

[0009] Processing the target remote sensing image through N local feature extraction layers of different scales in the CNN downsampling module to obtain N local feature maps of different scales, wherein any one of the local feature extraction layers has the same scale as the global feature extraction layer of the same number of layers in the multi-scale Transformer module;

[0010] Based on the same scale, the N global feature maps and the N local feature maps are respectively merged into N feature fusion maps;

[0011] Reconstructing the feature fusion map with the smallest scale among the N feature fusion maps into a target feature fusion map;

[0012] In accordance with the merging order from small to large scale, the target feature fusion map is merged with N-1 feature fusion maps other than the target feature fusion map in sequence to obtain an overall feature map, and a pixel-level segmentation result of the target remote sensing image is obtained based on the overall feature map.

[0013] In an exemplary embodiment, the first global feature extraction layer with the largest scale in the multi-scale Transformer module includes a first multi-scale convolutional layer, a first patch segmentation layer and a first Transformer layer, and the N-1 second global feature extraction layers in the multi-scale Transformer module except the first global feature extraction layer include a second multi-scale convolutional layer, a second patch segmentation layer, a patch merging layer and a second Transformer layer.

[0014] In an exemplary embodiment, the step of processing the target remote sensing image through the global feature extraction layers of N different scales in the multi-scale Transformer module to obtain global feature maps of N different scales comprises the following steps:

[0015] Inputting the target remote sensing image into the first multi-scale convolution layer in the first global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and fusing the target remote sensing image with the first initial multi-scale feature image into a first target multi-scale feature image;

[0016] Segmenting the first target multi-scale feature image into a plurality of first target multi-scale feature patch blocks by using the first patch segmentation layer;

[0017] Using the first Transformer layer to process the plurality of first target multi-scale feature patches to obtain a global feature map of the maximum scale output by the first global feature extraction layer;

[0018] Taking multiple first target multi-scale feature patches as input features, N-1 second global feature extraction layers are used to perform feature extraction steps in order from large to small scales to obtain N-1 global feature maps of different scales.

[0019] In an exemplary embodiment, the step of inputting the target remote sensing image into the first multi-scale convolution layer in the first global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and fusing the target remote sensing image with the first initial multi-scale feature image into a first target multi-scale feature image comprises the following steps:

[0020] Inputting the target remote sensing image into the first multi-scale convolutional layer in the first global feature extraction layer;

[0021] Performing a convolution operation on the target remote sensing image through a first hole convolution layer having a convolution kernel of a×a in the first multi-scale convolution layer to obtain a first feature image;

[0022] The target remote sensing image is convolved by a second hole convolution layer with a convolution kernel of b×b in the first multi-scale convolution layer to obtain a second feature image, a <b;

[0023] The target remote sensing image is convolved by a third hole convolution layer with a convolution kernel of c×c in the first multi-scale convolution layer to obtain a third feature image, b <c;

[0024] Performing an upsampling operation on the second feature image, and inputting the first feature image and the second feature image after the upsampling operation into the first dilated convolution layer to generate a multi-scale feature image;

[0025] Performing an upsampling operation on the multi-scale feature image, and inputting the third feature image and the multi-scale feature image after the upsampling operation into the first hole convolution layer to generate a first initial multi-scale feature image;

[0026] The target remote sensing image and the first initial multi-scale feature image are fused into a first target multi-scale feature image.

[0027] In an exemplary embodiment, the feature extraction step includes:

[0028] Performing a merging operation on the input features through the patch merging layer;

[0029] Inputting the combined input features into the second multi-scale convolution layer in the second global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a second initial multi-scale feature image, and fusing the input features with the second initial multi-scale feature image into a second target multi-scale feature image;

[0030] Segmenting the second target multi-scale feature image into a plurality of second target multi-scale feature patch blocks through the second patch segmentation layer, and using the plurality of second target multi-scale feature patch blocks as input features of the next second global feature extraction layer;

[0031] The second Transformer layer is used to process the plurality of second target multi-scale feature patch blocks to obtain a global feature map output by the second global feature extraction layer.

[0032] In an exemplary embodiment, the first Transformer layer includes a batch normalization layer, a compressed window self-attention mechanism layer, a feedforward network layer, and a compressed shift window self-attention mechanism layer.

[0033] In an exemplary embodiment, the key vector matrix and the numerical vector matrix of the attention mechanism in the compressed window self-attention mechanism layer and the compressed displacement window self-attention mechanism layer are compressed through a pooling operation.

[0034] In an exemplary embodiment, the step of processing the plurality of the first target multi-scale feature patches using the first Transformer layer to obtain the maximum-scale global feature map output by the first global feature extraction layer comprises the following steps:

[0035] A plurality of the first target multi-scale feature patch blocks are input into the first Transformer layer, and feature extraction processing is performed in sequence through the compression window self-attention mechanism layer, the feedforward network layer, the compression displacement window self-attention mechanism layer and the feedforward network layer to obtain the maximum scale global feature map output by the first global feature extraction layer, and normalization processing is performed through the batch normalization layer before each feature extraction processing.

[0036] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0037] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0038] Through the present invention, in the process of performing building segmentation on remote sensing images, the global feature map of the remote sensing image can be extracted through the multi-scale Transformer module, the local feature map of the remote sensing image can be extracted through the CNN downsampling module, and then the global feature map and the local feature map are combined and fused into an overall feature map, thereby achieving the effect of improving the accuracy of remote sensing image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a hardware structure block diagram of a mobile terminal according to an embodiment of the present invention;

[0040] Figure 2 is a flowchart of the steps of a remote sensing image building segmentation method based on a neural network model according to an embodiment of the present invention;

[0041] Figure 3 It is a framework flow chart of a remote sensing image building segmentation method based on a neural network model according to an embodiment of the present invention;

[0042] Figure 4 is a framework flow chart of a multi-scale convolutional layer according to an embodiment of the present invention;

[0043] Figure 5 is a framework flow chart of the Transformer layer according to an embodiment of the present invention;

[0044] Figure 6 1 is a schematic diagram of a process of compressing a key vector matrix and a numerical vector matrix in an attention mechanism through a pooling operation according to an embodiment of the present invention;

[0045] Figure 7 is a framework flow chart of a feedforward network layer according to an embodiment of the present invention;

[0046] Figure 8 Schematic diagram of the zero-filling process in the compressed shift window self-attention mechanism layer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0049] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal for a remote sensing image building segmentation method based on a neural network model according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a remote sensing image building segmentation method based on a neural network model in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0052] In this embodiment, a remote sensing image building segmentation method based on a neural network model is provided. Figure 2is a flow chart of a remote sensing image building segmentation method based on a neural network model according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0053] S110: Acquire a target remote sensing image to be processed.

[0054] In this step, the target remote sensing images are the input data for the segmentation model. These images can come from satellites, drones, or other remote sensing devices. The process of acquiring images can include preprocessing steps such as denoising, contrast enhancement, cropping, and normalization to ensure that the image quality is suitable for subsequent processing.

[0055] S120. Build a remote sensing image segmentation model by combining the multi-scale Transformer module and the CNN downsampling module.

[0056] In this step, refer to Figure 3 , building a hybrid neural network model that combines the features of Transformer and Convolutional Neural Network (CNN). The Transformer module is able to capture long-distance dependencies in the image, i.e. global features, while the CNN module is good at extracting local features. The multi-scale structure means that the model can process images at different resolutions, thereby capturing different levels of information.

[0057] S130, inputting the target remote sensing image into the remote sensing image segmentation model, processing the target remote sensing image through N global feature extraction layers of different scales in the multi-scale Transformer module, and obtaining N global feature maps of different scales.

[0058] In this step, if Figure 3 As shown, Figure 3 Where N is 4, the image is input to a multi-scale Transformer module, which contains multiple global feature extraction layers, each corresponding to a different scale. Each scale Transformer layer processes the image and outputs a global feature map, which captures the image context information at different scales.

[0059] S140, processing the target remote sensing image through N local feature extraction layers of different scales in the CNN downsampling module to obtain N local feature maps of different scales.

[0060] Similar to S130, such as Figure 3 As shown in , the image is also input into the CNN downsampling module, which contains 4 local feature extraction layers, each corresponding to a different scale. These layers extract local features of the image through convolution and pooling operations and output local feature maps. These feature maps capture the details of the image such as texture and edges. Figure 2 As shown in Figure 1, any local feature extraction layer has the same scale as the global feature extraction layer of the same number of layers in the multi-scale Transformer module.

[0061] S150, merging N global feature maps and N local feature maps into N feature fusion maps based on the same scale.

[0062] In this step, if Figure 3 As shown in Figure 1, the global feature map and the local feature map at each scale are merged, usually by concatenation or weighted summation. This can integrate global context information and local detail information to generate a richer feature representation.

[0063] S160: reconstruct the feature fusion map with the smallest scale among the N feature fusion maps into a target feature fusion map.

[0064] S170, merging the target feature fusion map with N-1 feature fusion maps other than the target feature fusion map in sequence according to the merging order from small to large scales to obtain an overall feature map, and obtaining a pixel-level segmentation result of the target remote sensing image based on the overall feature map.

[0065] In this step, if Figure 3 As shown in the figure, the feature fusion maps are gradually merged in order from small to large scales. This is usually done through upsampling and fusion operations, where upsampling refers to enlarging the low-resolution feature map to a higher resolution, and the fusion operation can be a simple addition or a more complex operation, such as using a convolutional layer to integrate features of different scales. This step-by-step merging method allows the model to take into account information from other scales at each scale, thereby generating an overall feature map that contains rich multi-scale information. This overall feature map will contain details and contextual information about the locations of each pixel in the remote sensing image, which is crucial for subsequent pixel-level classification. Finally, based on the overall feature map, the model will achieve the final pixel-level segmentation result through a linear mapping layer.

[0066] In one embodiment, Figure 3 As shown, the first global feature extraction layer with the largest scale in the multi-scale Transformer module includes a first multi-scale convolutional layer, a first patch segmentation layer and a first Transformer layer, and the N-1 second global feature extraction layers in the multi-scale Transformer module except the first global feature extraction layer include a second multi-scale convolutional layer, a second patch segmentation layer, a patch merging layer and a second Transformer layer. Figure 3 The second patch segmentation layer of all the second global feature extraction layers in are not shown.

[0067] In one embodiment, processing the target remote sensing image through the global feature extraction layers of N different scales in the multi-scale Transformer module to obtain the global feature maps of N different scales includes the following steps:

[0068] Inputting the target remote sensing image into the first multi-scale convolution layer in the first global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and fusing the target remote sensing image with the first initial multi-scale feature image into a first target multi-scale feature image;

[0069] Segmenting the first target multi-scale feature image into a plurality of first target multi-scale feature patch blocks through a first patch segmentation layer;

[0070] Using the first Transformer layer to process the plurality of first target multi-scale feature patches, obtaining a global feature map of the largest scale output by the first global feature extraction layer;

[0071] Taking multiple first target multi-scale feature patches as input features, N-1 second global feature extraction layers are used to perform feature extraction steps in descending order of scale to obtain N-1 global feature maps of different scales.

[0072] In this embodiment, a first initial multi-scale feature image is obtained by performing multi-way convolution using convolution kernels of different sizes. The convolution operation can extract local features of the image, and convolution kernels of different sizes can extract features of different scales, thereby forming a multi-scale feature image. Then, the original image is fused with the feature image extracted by convolution, usually by simple addition or other more complex fusion methods, such as convolution, full connection, etc., to obtain a target multi-scale feature image containing the original image information and local feature information.

[0073] Then, by sliding a fixed-size window on the feature image, the feature image is divided into multiple small blocks, that is, patches. The purpose of this is to be able to better utilize the Transformer module to process these patches and obtain global features in subsequent processing. Then, the first Transformer layer is used to process multiple first target multi-scale feature patch blocks to obtain the largest scale global feature map output by the first global feature extraction layer. The advantage of the Transformer module is that it can handle long-distance dependencies, which is especially important when processing remote sensing images, because one part of the image may affect other parts that are farther away. By processing multiple feature patch blocks, the Transformer module can obtain global context information and output a global feature map.

[0074] Finally, multiple first target multi-scale feature patches are used as input features, and N-1 second global feature extraction layers are used to perform feature extraction steps in order from large to small scales to obtain N-1 global feature maps of different scales. This step is to obtain global feature maps of different scales by performing global feature extraction at different scales. The purpose of this is to better integrate global features of different scales in subsequent processing, thereby obtaining more accurate remote sensing image segmentation results.

[0075] In one embodiment, the target remote sensing image is input into the first multi-scale convolution layer in the first global feature extraction layer, multi-way convolution is performed using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and the target remote sensing image and the first initial multi-scale feature image are fused into a first target multi-scale feature image, including the following steps:

[0076] Inputting the target remote sensing image into the first multi-scale convolutional layer in the first global feature extraction layer;

[0077] Performing a convolution operation on the target remote sensing image through a first hole convolution layer with a convolution kernel of a×a in the first multi-scale convolution layer to obtain a first feature image;

[0078] The second feature image is obtained by performing a convolution operation on the target remote sensing image through the second hole convolution layer with a convolution kernel of b×b in the first multi-scale convolution layer. <b;

[0079] The target remote sensing image is convolved by the third hole convolution layer with a convolution kernel of c×c in the first multi-scale convolution layer to obtain the third feature image, b <c;

[0080] Performing an upsampling operation on the second feature image, and inputting the first feature image and the second feature image after the upsampling operation into the first dilated convolutional layer to generate a multi-scale feature image;

[0081] Performing an upsampling operation on the multi-scale feature image, and inputting the third feature image and the multi-scale feature image after the upsampling operation into the first hole convolution layer to generate a first initial multi-scale feature image;

[0082] The target remote sensing image and the first initial multi-scale feature image are fused into a first target multi-scale feature image.

[0083] In this embodiment, the dilated convolution is also called the expanded convolution or the dilated convolution. The dilated convolution was originally proposed to solve the problem of image segmentation. Common image segmentation algorithms usually use pooling layers and convolution layers to increase the receptive field (Receptive Field), while also reducing the feature map size (resolution), and then use upsampling to restore the image size. The process of reducing and enlarging the feature map causes a loss in accuracy. Therefore, an operation is needed that can increase the receptive field while keeping the size of the feature map unchanged, thereby replacing the downsampling and upsampling operations. Under this requirement, the dilated convolution can be used. Unlike normal convolution, the dilated convolution introduces a hyper-parameter called "dilation rate", which defines the spacing between values ​​when the convolution kernel processes data. The dilation rate is also called the hole size.

[0084] The first atrous convolution layer, the second atrous convolution layer, and the third atrous convolution layer are all parallel atrous convolution layers in the first multi-scale convolution layer, and the dilation rate of the convolution kernel is 1. Figure 4 , assuming a=1, b=3, c=5, the three-way hole convolution layer simultaneously performs convolution operation on the target remote sensing image to extract the first feature image, the second feature image and the third feature image. The specific formula is as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] in, is the input feature (target remote sensing image), They represent the output results of the first dilated convolutional layer, the second dilated convolutional layer, and the third dilated convolutional layer respectively.

[0089] Then, the second feature image is upsampled, and the first feature image and the second feature image after the upsampling operation are input into the first hole convolution layer to generate a multi-scale feature image. The specific formula is as follows:

[0090] ;

[0091] in, is a multi-scale feature image, represents the addition operation, Indicates upsampling.

[0092] Then, the multi-scale feature image is upsampled, and the third feature image and the multi-scale feature image are input into the first hole convolution layer to generate the first initial multi-scale feature image. The specific formula is as follows:

[0093] ;

[0094] in, is the first initial multi-scale feature image, represents the addition operation, Indicates upsampling.

[0095] Finally, the target remote sensing image is fused with the first initial multi-scale feature image to form the first target multi-scale feature image. The specific formula is as follows:

[0096] ;

[0097] in, For the final output, For addition operation.

[0098] In one embodiment, the first Transformer layer includes a batch normalization layer, a compressed window self-attention mechanism layer, a feedforward network layer, and a compressed displacement window self-attention mechanism layer. The key vector matrix and the numerical vector matrix of the attention mechanism in the compressed window self-attention mechanism layer and the compressed displacement window self-attention mechanism layer are compressed by a pooling operation.

[0099] In this embodiment, the first Transformer layer is used to process a plurality of first target multi-scale feature patches to obtain a global feature map of the maximum scale output by the first global feature extraction layer, including the following steps:

[0100] Multiple first target multi-scale feature patch blocks are input into the first Transformer layer, and feature extraction is performed in sequence through the compressed window self-attention mechanism layer, the feedforward network layer, the compressed displacement window self-attention mechanism layer and the feedforward network layer to obtain the maximum scale global feature map output by the first global feature extraction layer. Before each feature extraction process, it is normalized through the batch normalization layer.

[0101] In this embodiment, refer to Figure 5 After the first target multi-scale feature patch blocks are input into the first Transformer layer, the first target multi-scale feature patch blocks are normalized by the batch normalization layer, and then the window attention is calculated by the compressed window self-attention mechanism layer. Figure 6As shown in the figure, based on the traditional Transformer, in order to reduce the calculation of large-scale multi-head attention, before the weight calculation, a pooling operation is used to reduce the key vector matrix and the numerical vector matrix of the attention mechanism in the compressed window self-attention mechanism layer, thereby reducing the resolution of the key vector matrix and the numerical vector matrix, and then reducing the amount of calculation. The specific process is to first use a 3×3 receptive field with a step size of 1 to shift within the matrix to divide the area. After each shift, the values ​​within the 3×3 range are randomly pooled to capture the local relationship of the input sequence and reduce the spatial dimension of the matrix. Finally, the processed feature information is connected through a residual connection to enhance the stability of the network. The specific formula is as follows:

[0102] ;

[0103] The features after the residual connection are further normalized by the batch normalization layer and then input into the feedforward network layer. In the traditional Transformer network, the operation of the multi-layer perceptron is too cumbersome. Here, we propose to use convolution to replace the traditional multi-layer perceptron to form a feedforward network. Figure 7 As shown in the figure, the input features are first fused through 1×1 convolution, then through two parallel 3×3 depth-separable convolutions, the two results are multiplied, and then through 1×1 convolution for overall compression and fusion, and finally residual connection is introduced to replace the multi-layer perceptron with this architecture. This design has a simple structure and low computational cost, and is an efficient feedforward network.

[0104] like Figure 7 As shown in the figure, the above process completes the processing of the first part of the first Transformer layer, and the processing result of the first part is sent to the compressed displacement window self-attention mechanism layer after being normalized by the batch normalization layer. When calculating the self-attention in this step, the same operation as the compressed window self-attention mechanism layer is used to reduce the dimension of the key vector matrix and the numerical vector matrix. In addition, in order to enhance the connection between windows and make up for the discontinuity between windows caused by window division, a filling sliding window self-attention mechanism is designed.

[0105] In the compressed window self-attention mechanism layer, the input feature X is divided into HW / M 2 In the compressed displacement window self-attention mechanism layer, the zero padding method is used to change the feature size. Figure 8 As shown in the figure, the zero-filling method is used to fill the periphery with zeros, and the original window size of M×M is adjusted to (M+2) 2, use the adjusted window to perform sliding partitions in the feature map, with a sliding step of M, and perform window self-attention calculation after each sliding window shift. This can enhance the information interaction between windows and make up for the information discontinuity caused by window segmentation. The output of the compressed displacement window self-attention mechanism layer is again connected and normalized through residual connection to enhance the stability of the network, and then passed through an efficient feedforward network to finally obtain the output of the first Transformer layer.

[0106] In one embodiment, the feature extraction step includes:

[0107] Merge the input features through the patch merging layer;

[0108] Inputting the combined input features to the second multi-scale convolution layer in the second global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a second initial multi-scale feature image, and fusing the input features with the second initial multi-scale feature image into a second target multi-scale feature image;

[0109] Segmenting the second target multi-scale feature image into a plurality of second target multi-scale feature patch blocks through a second patch segmentation layer, and using the plurality of second target multi-scale feature patch blocks as input features of a next second global feature extraction layer;

[0110] The second Transformer layer is used to process the multiple second target multi-scale feature patches to obtain a global feature map output by the second global feature extraction layer.

[0111] In this embodiment, the second multi-scale convolution layer in the second global feature extraction layer has the same structure as the first multi-scale convolution layer, and the steps for processing the input features are also the same. The second Transformer layer in the second global feature extraction layer has the same structure as the first Transformer layer, and the steps for processing the input features are also the same.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0113] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0114] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0115] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0116] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0117] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0118] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing image building segmentation method based on a neural network model, characterized in that: include: Acquire the target remote sensing image to be processed; Combine the multi-scale Transformer module and the CNN downsampling module to build a remote sensing image segmentation model; Inputting the target remote sensing image into the remote sensing image segmentation model, processing the target remote sensing image through N global feature extraction layers of different scales in the multi-scale Transformer module, and obtaining N global feature maps of different scales; Processing the target remote sensing image through N local feature extraction layers of different scales in the CNN downsampling module to obtain N local feature maps of different scales, wherein any one of the local feature extraction layers has the same scale as the global feature extraction layer of the same number of layers in the multi-scale Transformer module; Based on the same scale, the N global feature maps and the N local feature maps are respectively merged into N feature fusion maps; Reconstructing the feature fusion map with the smallest scale among the N feature fusion maps into a target feature fusion map; According to the merging order from small to large scale, the target feature fusion map is merged with N-1 feature fusion maps other than the target feature fusion map in sequence to obtain an overall feature map, and a pixel-level segmentation result of the target remote sensing image is obtained based on the overall feature map; The first global feature extraction layer with the largest scale in the multi-scale Transformer module includes a first multi-scale convolutional layer, a first patch segmentation layer and a first Transformer layer, and the N-1 second global feature extraction layers in the multi-scale Transformer module except the first global feature extraction layer include a second multi-scale convolutional layer, a second patch segmentation layer, a patch merging layer and a second Transformer layer; The step of processing the target remote sensing image through the global feature extraction layers of N different scales in the multi-scale Transformer module to obtain global feature maps of N different scales includes the following steps: Inputting the target remote sensing image into the first multi-scale convolution layer in the first global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and fusing the target remote sensing image with the first initial multi-scale feature image into a first target multi-scale feature image; Segmenting the first target multi-scale feature image into a plurality of first target multi-scale feature patch blocks by using the first patch segmentation layer; Using the first Transformer layer to process the plurality of first target multi-scale feature patches to obtain a global feature map of the maximum scale output by the first global feature extraction layer; Taking multiple first target multi-scale feature patches as input features, N-1 second global feature extraction layers are used to perform feature extraction steps in order from large to small scales to obtain N-1 global feature maps of different scales.

2. The method according to claim 1, characterized in that The step of inputting the target remote sensing image into the first multi-scale convolution layer in the first global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a first initial multi-scale feature image, and fusing the target remote sensing image with the first initial multi-scale feature image into a first target multi-scale feature image comprises the following steps: Inputting the target remote sensing image into the first multi-scale convolutional layer in the first global feature extraction layer; Performing a convolution operation on the target remote sensing image through a first hole convolution layer having a convolution kernel of a×a in the first multi-scale convolution layer to obtain a first feature image; The target remote sensing image is convolved by a second hole convolution layer with a convolution kernel of b×b in the first multi-scale convolution layer to obtain a second feature image, a <b; The target remote sensing image is convolved by a third hole convolution layer with a convolution kernel of c×c in the first multi-scale convolution layer to obtain a third feature image, b <c; Performing an upsampling operation on the second feature image, and inputting the first feature image and the second feature image after the upsampling operation into the first dilated convolution layer to generate a multi-scale feature image; Performing an upsampling operation on the multi-scale feature image, and inputting the third feature image and the multi-scale feature image after the upsampling operation into the first hole convolution layer to generate a first initial multi-scale feature image; The target remote sensing image and the first initial multi-scale feature image are fused into a first target multi-scale feature image.

3. The method according to claim 1, characterized in that The feature extraction step comprises: Performing a merging operation on the input features through the patch merging layer; Inputting the combined input features into the second multi-scale convolution layer in the second global feature extraction layer, performing multi-way convolution using convolution kernels of different sizes to obtain a second initial multi-scale feature image, and fusing the input features with the second initial multi-scale feature image into a second target multi-scale feature image; Segmenting the second target multi-scale feature image into a plurality of second target multi-scale feature patch blocks through the second patch segmentation layer, and using the plurality of second target multi-scale feature patch blocks as input features of the next second global feature extraction layer; The second Transformer layer is used to process the plurality of second target multi-scale feature patch blocks to obtain a global feature map output by the second global feature extraction layer.

4. The method according to claim 1, characterized in that: The first Transformer layer includes a batch normalization layer, a compressed window self-attention mechanism layer, a feedforward network layer and a compressed shift window self-attention mechanism layer.

5. The method according to claim 4, characterized in that The key vector matrix and the numerical vector matrix of the attention mechanism in the compressed window self-attention mechanism layer and the compressed displacement window self-attention mechanism layer are compressed through pooling operations.

6. The method according to claim 5, characterized in that The step of processing the plurality of the first target multi-scale feature patches by using the first Transformer layer to obtain the global feature map of the maximum scale output by the first global feature extraction layer comprises the following steps: A plurality of the first target multi-scale feature patch blocks are input into the first Transformer layer, and feature extraction processing is performed in sequence through the compression window self-attention mechanism layer, the feedforward network layer, the compression displacement window self-attention mechanism layer and the feedforward network layer to obtain the maximum scale global feature map output by the first global feature extraction layer, and normalization processing is performed through the batch normalization layer before each feature extraction processing.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images