A semantic segmentation method, system, device and medium

Abstract

The application discloses a semantic segmentation method, system, device and medium, the method comprises the following steps: inputting an input image into a ResNet50 main encoder and a swin-transformer auxiliary encoder based on a detail structure maintaining algorithm and a spatial feature enhancement algorithm for feature extraction; the extracted features are fused by a position perception attention module, and the fused features are upsampled by a U-shaped structure to obtain a segmentation result; the application can collect context information of different spatial dimensions in the encoding stage based on the detail structure maintaining algorithm and the spatial feature enhancement algorithm, alleviate semantic ambiguity, realize effective combination of global information and local information based on the position perception attention mechanism, conform to the cognitive mode of remote sensing images, and comprehensively improve the accuracy of semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and particularly to a semantic segmentation method, system, device and medium. BACKGROUND

[0002] Image semantic segmentation technology is an artificial intelligence-based recognition algorithm, and remote sensing image semantic segmentation technology can analyze the features and changes of the earth's surface, and has important roles in environmental monitoring, urban planning and natural resource management. Specifically, the technology can identify buildings, roads, water bodies and greenery, etc. urban elements, providing scientific basis for urban planning and management, and can also be used for disaster monitoring and response. Remote sensing image semantic segmentation technology has a wide application prospect, and provides strong support for fully understanding and utilizing earth resources, protecting the environment and building cities, etc.

[0003] At present, deep learning methods have the advantages of automatic learning of features and models, and have become the mainstream method of remote sensing image semantic segmentation. For image semantic segmentation problems, many deep learning models have been developed in the prior art, such as fully convolutional networks, deep cascaded convolutional neural networks and multi-scale context networks. These models can be applied to different image datasets through pre-training and fine-tuning, and good results have been achieved.

[0004] However, although these deep convolutional neural network-based models have achieved considerable results on other datasets, they are still limited by convolution kernels and can only extract local information, lacking the ability to model contextual information. Remote sensing image scenes are complex, and objects within the same class may have large feature differences, while objects between different classes may have similar features. It is currently difficult for deep convolutional neural network-based methods to achieve optimal / accurate results on remote sensing image semantic segmentation tasks.

[0005] In addition, the visual Transformer architecture in the prior art relies on its global attention mechanism to capture long-range dependencies between patches, and performs particularly well in detection, segmentation and other visual tasks that require a large receptive field. Currently, this architecture is also being studied for semantic segmentation tasks. However, it should be noted that when humans face complex scenes and diverse ground objects in remote sensing images, they will understand and interpret the distribution and spatial structure of the ground objects in the image through a local and global understanding method. Local detection can identify details and local features in the image (such as the roof of a building, the texture of a road, and the shape of a tree, etc.), and global detection captures the overall distribution and structure in the image (such as the layout of urban buildings, the distribution of water bodies, and forest coverage). For the transformer method, when it is applied to natural language processing tasks, the input representation of the visual task is more complex due to its unique 2D structure, and the local spatial information between patches is also very important. In the process of patching the input image, the transformer method will destroy the internal structure information of the image block, and the long-range attention mechanism is easy to ignore the local unique properties of the image, so it does not conform to the cognitive method of remote sensing image semantic segmentation and cannot achieve accurate segmentation.

[0006] In summary, the scene of remote sensing images is complex, the model based on deep convolutional neural network is limited by the convolution kernel, lacks the ability to model context information, and the transformer method ignores the local features of the image. Both of the above methods are difficult to accurately segment the semantics of remote sensing images. SUMMARY

[0007] The purpose of the present application is to provide a semantic segmentation method, system, device and medium to solve the problem that the prior art semantic segmentation method cannot accurately segment the semantics of remote sensing images.

[0008] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0009] In one aspect, the present application provides a semantic segmentation method, comprising the following steps:

[0010] Feature extraction step: input the input image into the ResNet50 main encoder and the swin-transformer auxiliary encoder based on the detail structure preservation algorithm and the spatial feature enhancement algorithm for feature extraction;

[0011] Feature fusion step: the extracted features are fused by a position-aware attention module, and the fused features are upsampled by a U-shaped structure to obtain a segmentation result.

[0012] As an improved scheme, the feature extraction step of the input image in the ResNet50 main encoder includes:

[0013] The size of the input image is compressed by half and sent into the ResNet50 main encoder;

[0014] The ResNet50 network of the ResNet50 main encoder is used to perform four-stage feature extraction on the input image;

[0015] The output size of the ResNet50 network is Wherein, Z n The output size of the ResNet50 network is R, H represents the length of the input image R, W represents the width of the input image R, and C2=128.

[0016] As an improved scheme, the feature extraction step of the input image in the swin-transformer auxiliary encoder includes:

[0017] The input image is sent into the swin-transformer auxiliary encoder for four-stage feature extraction;

[0018] In the feature extraction process of the swin-transformer auxiliary encoder, when the patches are down-sampled, first, the first feature enhancement processing is performed through the detail structure maintaining algorithm, and then the second feature enhancement processing is performed through the spatial feature enhancement algorithm;

[0019] The output size of the swin-transformer auxiliary encoder is Wherein, S n The output size of the swin-transformer auxiliary encoder is R, H represents the length of the input image R, W represents the width of the input image R, and C1=96.

[0020] As an improved scheme, the swin-transformer auxiliary encoder includes a first self-attention module and a second self-attention module; the first self-attention module is a W-Trans block, and the second self-attention module is a SW-Trans block;

[0021] In each stage of feature extraction of the input image by the swin-transformer auxiliary encoder, the first self-attention module and the second self-attention module are alternately executed;

[0022] The principle formula of the alternately executed includes:

[0023]

[0024]

[0025]

[0026]

[0027] wherein, and are the outputs of the first self-attention module and the second self-attention module, respectively; s l and s l+1 are the outputs after passing through the MLP layer.

[0028] As an improved scheme, the detail structure maintaining algorithm comprises:

[0029] The vector is reshaped into a feature map;

[0030] The reshaped feature map is respectively put into a 3x3 dilated convolution and a 1x1 convolution layer to obtain a first result and a second result;

[0031] The first result and the second result are added through a GELU activation function;

[0032] The added result is combined with the original feature map using a residual link, and is reshaped into a vector form;

[0033] The mathematical expression of the detail structure maintaining algorithm is:

[0034] out=GELU(DConv(x)+conv(x))+x.

[0035] As an improved scheme, the spatial feature enhancement algorithm comprises:

[0036] A strip-shaped convolution is adopted to collect remote context information in two spatial dimensions based on a pooling operation;

[0037] A convolution operation is performed, and the features of the two dimensions are added;

[0038] The mathematical expression of the spatial feature enhancement algorithm is:

[0039]

[0040]

[0041] wherein:

[0042] i, j are indexes in vertical and horizontal directions respectively, 0<=i<h, 0<=j<w;

[0043] Convh is a convolution with a kernel of (3, 1), a stride of 1 and zero padding of (1, 0);

[0044] Convw is a convolution with a kernel of (1, 3), a stride of 1 and zero padding of (0, 1);

[0045] f is an up-sampling operation;

[0046] φ is a 1x1 convolution operation with a Relu activation function.

[0047] As an improved scheme, the feature fusion processing comprises:

[0048] performing feature fusion processing based on a position-aware attention mechanism according to a feature fusion formula;

[0049] The feature fusion formula comprises:

[0050]

[0051]

[0052] Wherein:

[0053] CA is an attention mechanism;

[0054] X is matrix multiplication;

[0055] Conv is a 3x3 convolution;

[0056] Z i is the output of the main encoder feature extraction stage;

[0057] S i is the output of the auxiliary encoder feature extraction stage;

[0058] f i is the finally generated feature.

[0059] In another aspect, the present application also provides a semantic segmentation system, comprising:

[0060] a feature extraction module and a feature fusion module;

[0061] The feature extraction module is configured to input an input image into a ResNet50 main encoder and a swin-transformer auxiliary encoder based on a detail structure preservation algorithm and a spatial feature enhancement algorithm for feature extraction.

[0062] The feature fusion module is configured to perform feature fusion processing on the extracted features through a position-aware attention module, and perform up-sampling on the fused features through a U-shaped structure to obtain a segmentation result.

[0063] In another aspect, the present application also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the steps of the semantic segmentation method.

[0064] In another aspect, the present application also provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus; and wherein:

[0065] The memory is configured to store a computer program.

[0066] The processor is configured to execute the steps of the semantic segmentation method by running the program stored in the memory.

[0067] The technical scheme of the present application has the following advantages:

[0068] 1. The semantic segmentation method can realize the detail structure preserving algorithm and the spatial feature enhancement algorithm, collect context information of different spatial dimensions in the encoding stage, alleviate semantic ambiguity, realize effective combination of global information and local information based on the position-aware attention mechanism, conform to the cognitive mode of remote sensing images, and improve the accuracy of semantic segmentation in all aspects.

[0069] 2. The semantic segmentation system can realize the detail structure preserving algorithm and the spatial feature enhancement algorithm through the cooperation of the feature extraction module and the feature fusion module, collect context information of different spatial dimensions in the encoding stage, alleviate semantic ambiguity, realize effective combination of global information and local information based on the position-aware attention mechanism, conform to the cognitive mode of remote sensing images, and improve the accuracy of semantic segmentation in all aspects.

[0070] 3. The computer readable storage medium can realize the cooperation of the feature extraction module and the feature fusion module, and realize the semantic segmentation method.

[0071] 4. The computer device can store and execute the computer readable storage medium, and realize the semantic segmentation method. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings required to be used in the description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0073] Figure 1 is a flowchart of the semantic segmentation method described in Embodiment 1 of the present application;

[0074] Figure 2 is a detailed flowchart of the semantic segmentation method described in Embodiment 1 of the present application;

[0075] Figure 3 is a schematic diagram of the logical architecture of STU-net in the semantic segmentation method described in Embodiment 1 of the present application;

[0076] Figure 4 is a schematic diagram of the architecture of the detail structure preservation algorithm in the semantic segmentation method described in Embodiment 1 of the present application;

[0077] Figure 5 is a schematic diagram of the architecture of the spatial feature enhancement algorithm in the semantic segmentation method described in Embodiment 1 of the present application;

[0078] Figure 6 is a schematic diagram of the architecture of the position-aware attention mechanism in the semantic segmentation method described in Embodiment 1 of the present application;

[0079] Figure 7 is a schematic diagram of the application effect of the detail structure preservation algorithm in the semantic segmentation method described in Embodiment 1 of the present application;

[0080] Figure 8 is a schematic diagram of the application effect of the spatial feature enhancement algorithm in the semantic segmentation method described in Embodiment 1 of the present application;

[0081] Figure 9 is a schematic diagram of the application effect of the FFM framework in the semantic segmentation method described in Embodiment 1 of the present application;

[0082] Figure 10 is a schematic diagram of the architecture of the semantic segmentation system described in Embodiment 2 of the present application;

[0083] Figure 11 is a schematic diagram of the structure of the computer device described in Embodiment 4 of the present application;

[0084] The markers in the drawings are explained as follows:

[0085] a1, application effect of the reference processing method; b1, application effect after using the detail structure preservation algorithm;

[0086] a2, application effect of the benchmark processing method; b2, application effect after using the spatial feature enhancement algorithm;

[0087] a3, application effect of the benchmark processing method; b3, application effect after using the FFM framework;

[0088] 1501, processor; 1502, communication interface; 1503, memory; 1504, communication bus. DETAILED DESCRIPTION

[0089] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the scope of protection of the present application can be more clearly defined.

[0090] In the description of the present application, it should be noted that the embodiments described in the present application are part of the embodiments of the present application, not all the embodiments; all other embodiments obtained by those skilled in the art without creative labor based on the embodiments in the present application, belong to the scope of protection of the present application.

[0091] The terms "first", "second", and the like in the specification and claims herein and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0092] Embodiment 1

[0093] The present embodiment provides a semantic segmentation method, as shown in Figures 1-9 , including the following steps:

[0094] As an embodiment of the present application, the method is based on STU-net for semantic segmentation processing, wherein, as shown in Figure 3 , the STU-net has a skip encoder-decoder structure, and a hybrid model of swin-transformer and CNN is used in the decoder part, and finally the features of the two encoder branches are fused through a feature fusion module and sent into the main encoder;

[0095] As an embodiment of the present application, the main steps of the method specifically include:

[0096] S100, input the input image into a ResNet50 main encoder and a swin-transformer auxiliary encoder based on a detail structure maintaining algorithm and a spatial feature enhancing algorithm respectively for feature extraction;

[0097] S200, performing feature fusion processing on the extracted features by a position-aware attention module, and obtaining an output (segmentation) result by upsampling the fused features through a U-shaped structure.

[0098] As an embodiment of the present application, in step S100, specifically comprising:

[0099] First, a given input image X∈R H*W*C is input into a ResNet50 main encoder and a swin-transformer auxiliary encoder respectively; wherein H represents the length of the input image, W represents the width of the input image, and C represents the channel number of the input image;

[0100] As a preferred embodiment, when the input image is input into the swin-transformer auxiliary encoder for feature extraction, the following operations are performed:

[0101] Specifically, in the swin-transformer auxiliary encoder, the size of the patch is set to 8x8, and the repetition rate is maintained at 50%; these patches need to be projected into C1 dimension, and then put into the swin-transformer block;

[0102] Specifically, the swin-transformer block is composed of two self-attention modules W-MSA (renamed as W-Trans block in this method) and SW-MSA (renamed as SW-Trans block in this method); and the standard transformer block is composed of multi-head sub-attention, multi-layer perception and layer normalization; wherein the multi-head sub-attention needs to perform self-attention calculation on one token and all other tokens, and the calculation amount of the model grows in quadratic form, which is too expensive for dense task semantic segmentation; when calculating, the image needs to be divided into non-overlapping windows, and self-attention calculation is performed within the window, and each window contains MxM patches;

[0103] And in the method, before each patch token enters the swin-transformer block, it is first put into the detail-structure preservation algorithm; then the obtained vector is first passed through the spatial feature enhancement algorithm, and then connected to the vector through the swin-transformer block; in the embodiment, the swin-transformer auxiliary encoder has four feature extraction stages, and the outputs of the four stages are defined as S n , n = 1, 2, 3, 4; the output size is, Where C1 = 96;

[0104] Specifically, in each stage of feature extraction of the auxiliary encoder in the method, two W-Trans blocks and SW-Trans blocks are alternately executed, and the specific formula is as follows:

[0105]

[0106]

[0107]

[0108]

[0109] In the above execution formula:

[0110] And are the outputs of W-MSA and SW-MSA respectively; s l and s l+1 are the outputs through the MLP layer;

[0111] Specifically, in the swin-transformer, the model reduces the patches by projection; after the down-sampling process, the detail-structure preservation algorithm is introduced to improve the segmentation accuracy; as shown in Figure 4 , the detail-structure preservation algorithm is as follows: first, the vector is reshaped into a feature map form, then the feature map is respectively put into a 3x3 dilated convolution (where the dilation factor = 2) and a 1x1 convolution layer, then added through a GELU activation function and combined with the original feature map using a residual link, and finally reshaped back to a vector form; Specifically, the mathematical expression of the detail-structure preservation algorithm is: out = GELU(DConv(x) + conv(x)) + x;

[0112] Specifically, as shown in Figure 5 , the spatial feature enhancement algorithm is: using strip convolution to collect long-range context information in two spatial dimensions; reshape into g i, j ∈R (c×h×w) ; wherein c=2 n-1 C1,h=H / 2 n-1 ,w=W / 2 n-1 ; g i , j The pooling operation is performed along the vertical and horizontal dimensions respectively, and the context information of different spatial dimensions is collected through the pooling convolution operation in the horizontal and vertical directions to relieve semantic ambiguity. Then, the convolution operation is performed, and finally, the features of the two dimensions are added. The mathematical expression of the algorithm is as follows:

[0113]

[0114]

[0115] In the above formula:

[0116] i, j are indexes in the vertical and horizontal directions respectively, 0<=i<h, 0<=j<w;

[0117] Convh represents a convolution kernel (3, 1) with a step size of 1 and zero padding (1, 0);

[0118] Convw represents a convolution kernel (1, 3) with a step size of 1 and zero padding (0, 1);

[0119] f is an up-sampling operation;

[0120] φ is a 1x1 convolution operation with a Relu activation function;

[0121] As a preferred embodiment, when the input image is fed into the ResNet50 main encoder for feature extraction, the following operations are performed:

[0122] The image size is compressed by half and fed into the ResNet50 network. The main encoder has four feature extraction stages, and the output definitions of the four stages are Z n , n=1, 2, 3, 4; the output size is: wherein C2=128; it can be known from Figure 3 that the FFM framework is also set in the ResNet50 main encoder, which can further improve the segmentation accuracy;

[0123] As an embodiment of the present application, in step S200, it specifically comprises:

[0124] Specifically, in the present embodiment, the features of the main encoder and the auxiliary encoder are fused together through the residual connection feature fusion module based on Coord Attention and put into the main encoder.

[0125] Specifically, in the aforementioned encoding stage, the final output result will be obtained After that, the resolution is expanded through the deconvolution layer, and the features of the decoder and the encoder are connected through the skip connection layer; finally, 3x3 convolution and linear interpolation up-sampling are used to predict the corresponding results;

[0126] Specifically, in the above operation, in order to effectively fuse the features from the ResNet34 main encoder and the swin-transformer auxiliary encoder, the method uses a new feature fusion module, such as Figure 6 As shown, that is, the feature fusion module based on position-aware attention mechanism, which uses position-aware channel attention mechanism to accurately locate and identify the target of interest, and then strengthens the global and local information obtained by the two encoders, and finally embeds the processed features into the features obtained by the encoder;

[0127] Specifically, the feature fusion module based on position-aware attention mechanism has the following feature fusion formula:

[0128]

[0129]

[0130] In the above feature fusion formula:

[0131] CA, for Coordinate Attention;

[0132] x, for matrix multiplication;

[0133] Conv, for 3x3 convolution;

[0134] Z i , for the output of the main encoder feature extraction stage;

[0135] S i , for the output of the auxiliary encoder feature extraction stage;

[0136] f i , for the final generated feature, which effectively combines global information and local information.

[0137] Based on the method, the feature extraction in the encoder stage can be enhanced, the context information can be combined, and the semantic segmentation accuracy can be improved.

[0138] Embodiment 2

[0139] This embodiment is based on the same inventive concept as the semantic segmentation method described in embodiment 1, and provides a semantic segmentation system, as shown in Figure 10As shown, it comprises a feature extraction module and a feature fusion module.

[0140] The feature extraction module is configured to input the input image into a ResNet50 main encoder and a swin-transformer auxiliary encoder based on a detail structure maintaining algorithm and a spatial feature enhancement algorithm for feature extraction.

[0141] As an implementation form, the feature extraction module compresses the size of the input image by half and sends it into the ResNet50 main encoder; the feature extraction module uses the ResNet50 network of the ResNet50 main encoder to perform four-stage feature extraction on the input image; the output size of the ResNet50 network is wherein Z n is the output size of the ResNet50 network, R is the input image, H represents the length of the input image R, W represents the width of the input image R, and C2=128.

[0142] As an implementation form, the feature extraction module sends the input image into the swin-transformer auxiliary encoder for four-stage feature extraction; in the feature extraction process of the swin-transformer auxiliary encoder, when the patches are down-sampled, first, the first feature enhancement processing is performed through the detail structure maintaining algorithm, and then the second feature enhancement processing is performed through the spatial feature enhancement algorithm; the output size of the swin-transformer auxiliary encoder is wherein S n is the output size of the swin-transformer auxiliary encoder, R is the input image, H represents the length of the input image R, W represents the width of the input image R, and C1=96.

[0143] As an implementation form, the detail structure maintaining algorithm comprises: remodeling a vector into a feature map; putting the remodeled feature map into a 3x3 dilated convolution and a 1x1 convolution layer respectively to obtain a first result and a second result; adding the first result and the second result through a GELU activation function; combining the added result with the original feature map using a residual link and remodeling it into a vector form; the mathematical expression of the detail structure maintaining algorithm is: out=GELU(DConv(x)+conv(x))+x.

[0144] As an implementation form, the spatial feature enhancement algorithm comprises: collecting remote context information based on a pooling operation in two spatial dimensions by adopting a strip-shaped convolution; performing a convolution operation and adding features in two dimensions; and a mathematical expression of the spatial feature enhancement algorithm is:

[0145] wherein i and j are indexes in the vertical direction and the horizontal direction respectively, 0<=i

[0146] The feature fusion module is configured to perform feature fusion processing on the extracted features by a position-aware attention module, and perform up-sampling on the fused features by a U-shaped structure to obtain a segmentation result.

[0147] As an implementation form, the feature fusion processing comprises: the feature fusion module performs feature fusion processing based on a position-aware attention mechanism according to a feature fusion formula; and the feature fusion formula comprises: wherein CA is an attention mechanism, x is matrix multiplication, Conv is a 3x3 convolution, and Z i is an output of a main encoder feature extraction stage; S i is an output of an auxiliary encoder feature extraction stage; f i is a finally generated feature.

[0148] Embodiment 3

[0149] The embodiment provides a computer-readable storage medium comprising:

[0150] The storage medium is used to store computer software instructions for implementing the semantic segmentation method described in Embodiment 1, which contains an executable program for performing the program set for the semantic segmentation method. Specifically, the executable program can be built into the semantic segmentation system described in Embodiment 2, so that the semantic segmentation system can implement the semantic segmentation method described in Embodiment 1 by executing the built-in executable program.

[0151] In addition, the computer-readable storage medium of the embodiment can adopt any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any combination thereof.

[0152] Embodiment 4

[0153] The embodiment provides an electronic device, such as Figure 11 As shown in the figure, the electronic device can include a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete mutual communication through the communication bus 1504.

[0154] The memory 1503 is used for storing a computer program.

[0155] The processor 1501 is used for executing the computer program stored in the memory 1503, and the steps of the semantic segmentation method described in the above embodiment 1 are realized.

[0156] As an embodiment of the present application, the communication bus mentioned by the terminal can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 11 In the figure, only one thick line is used to represent the communication bus, but it does not mean that there is only one bus or only one type of bus.

[0157] As an embodiment of the present application, the communication interface is used for communication between the terminal and other devices.

[0158] As an embodiment of the present application, the memory can include a random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0159] As an embodiment of the present application, the processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0160] Compared with the prior art, the semantic segmentation method, system, device and medium provided by the present application can collect context information of different spatial dimensions in the encoding stage based on a detail structure preservation algorithm and a spatial feature enhancement algorithm, alleviate semantic ambiguity, realize effective combination of global information and local information based on a position-aware attention mechanism, conform to the cognitive mode of remote sensing images, and improve the accuracy of semantic segmentation in all aspects.

[0161] It should be understood that, in various embodiments herein, the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments herein.

[0162] It should also be understood that, in the embodiments herein, the term "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0163] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0165] In several embodiments provided herein, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic, and the division of the units is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electric, mechanical or in other forms.

[0166] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments provided herein.

[0167] In addition, each functional unit in each embodiment herein can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0168] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions herein, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment herein. The foregoing storage medium includes various media that can store program codes, such as U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0169] The above merely illustrates the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which are made by using the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A method of semantic segmentation, characterized in that, The method comprises the following steps: The feature extraction step: input the input image into the ResNet50 main encoder and the swin-transformer auxiliary encoder based on the detail structure preservation algorithm and the spatial feature enhancement algorithm for feature extraction; The feature fusion step: the extracted features are fused by the position-aware attention module, and the fused features are up-sampled by the U-shaped structure to obtain the segmentation result; The feature extraction step of the input image in the swin-transformer auxiliary encoder includes: feeding the input image into the swin-transformer auxiliary encoder for four-stage feature extraction; in the feature extraction process of the swin-transformer auxiliary encoder, when the patches are down-sampled, first, the first feature enhancement processing is performed through the detail structure maintaining algorithm, and then the second feature enhancement processing is performed through the spatial feature enhancement algorithm; the output size of the swin-transformer auxiliary encoder is wherein, is the output size of the swin-transformer auxiliary encoder, R is an input image, H represents the length of the input image R, and W represents the width of the input image R, n=1, 2, 3, 4; The detailed structure preservation algorithm includes: reshaping the vector into a feature map; and placing the reshaped feature map into... dilated convolution and In the convolutional layer, a first result and a second result are obtained; the first result and the second result are added together using the GELU activation function; the added result is combined with the original feature map using residual links and reshaped into a vector form; the mathematical expression of the detail structure preservation algorithm is: +x ; The spatial feature enhancement algorithm comprises: collecting remote context information in two spatial dimensions based on the pooling operation by using the strip convolution; performing convolution operation and adding the features of the two dimensions; The mathematical expression of the spatial feature enhancement algorithm is ; ; wherein: the formula is reshaped into , h = H / 2 n-1 , w = W / 2 n-1 ; i, j are indexes in the vertical direction and the horizontal direction respectively, 0<=i is a convolution kernel (3, 1) with a step of 1 and zero padding (1, 0); is a convolution kernel (1, 3) with a step of 1 and zero padding (0, 1); f is an up-sampling operation; and φ is a 1x1 convolution operation with a Relu activation function.​ The feature fusion processing comprises: performing feature fusion processing based on the position-aware attention mechanism according to the feature fusion formula; The feature fusion formula comprises: ; Residual [ , , ]);Wherein: is an attention mechanism; is matrix multiplication; is a 3x3 convolution; is the output of the main encoder feature extraction stage; is the output of the auxiliary encoder feature extraction stage; is the final generated feature.

2. The semantic segmentation method according to claim 1, wherein: The feature extraction step of the input image in the ResNet50 main encoder comprises: The size of the input image is compressed by half and sent into the ResNet50 main encoder; The ResNet50 network of the ResNet50 main encoder is used to extract features of the input image in four stages; The output size of the ResNet50 network is wherein, is the output size of the ResNet50 network, R is an input image, H represents the length of the input image R, W represents the width of the input image R, .

3. The semantic segmentation method according to claim 1, wherein: The swin-transformer auxiliary encoder comprises a first self-attention module and a second self-attention module; the first self-attention module is a W-Trans block, and the second self-attention module is a SW-Trans block; In each stage of feature extraction of the swin-transformer auxiliary encoder, the first self-attention module and the second self-attention module are alternately executed; The principle formula of the alternately executed comprises: ； ； ； ； wherein, and are the outputs of the first and second self-attention modules, respectively; and are the outputs of the MLP layers.

4. A semantic segmentation system based on the semantic segmentation method according to any one of claims 1 to 3, characterized in that, The system comprises a feature extraction module and a feature fusion module; The feature extraction module is configured to input the input image into the ResNet50 main encoder and the swin-transformer auxiliary encoder based on the detail structure preservation algorithm and the spatial feature enhancement algorithm for feature extraction; The feature fusion module is configured to fuse the extracted features by the position-aware attention module, and up-sample the fused features by the U-shaped structure to obtain the segmentation result.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the semantic segmentation method in any one of claims 1-3.

6. A computer device, comprising: The system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory can communicate with each other through the communication bus; wherein: The memory is configured to store a computer program; The processor is configured to execute the steps of the semantic segmentation method in any one of claims 1-3 by running the program stored in the memory.

Images