Remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network

Abstract

The invention relates to the technical field of deep learning, and proposes a remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network, and the method employs an improved EffcientNetV2 as a backbone network, and LMFF-Net remarkably reduces the demands of computing resources, and improves the extraction capability of multi-scale features. By integrating the channel attention mechanism, the LMFF-Net enhances the recognition and attention of the key features of the ground feature edge. Besides, a new PMU module is researched and developed, the fusion effect of multi-scale features is optimized, and by improving a DenseAspp module, the capability of capturing details of ground object targets with different scales is further improved. The invention provides a new method for deeply researching a deep learning technology in remote sensing image land utilization and land cover information extraction, and provides a new method reference for remote sensing image land utilization and land cover information extraction.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network. Background of the Invention

[0003] The rapid development of remote sensing and computer deep learning technology has provided key technical support for the rapid extraction of regional land use and land cover information. How to realize the intelligent extraction of land use and land cover information from long-term remote sensing images is a hot issue in the current research on the application of remote sensing science and technology. The research and application of intelligent extraction technology of land use and land cover information based on remote sensing images has important theoretical and practical significance for promoting the rational use and sustainable development of land resources and promoting the application of remote sensing science and technology.

[0004] Due to the large size and high dimensionality of remote sensing image data, traditional convolutional neural networks require deeper and more complex network structures to accurately extract detailed feature information of objects when extracting land use and land cover information, which increases the complexity of convolutional neural network model design and the difficulty of training. At the same time, deeper convolutional neural network structures require longer training and reasoning time, which will cause memory resource consumption to increase exponentially and require a large amount of computing resources. Lightweight convolutional neural networks can ensure high performance while consuming less resources, thus playing an important role in land use and land cover information extraction. Therefore, how to design an effective lightweight convolutional neural network with limited computing resources to quickly and efficiently extract land use and land cover information from remote sensing images has very important practical research value.

[0005] When extracting land use and land cover information from remote sensing images, remote sensing images contain a large amount of detailed information about objects. There are many types of objects, and the scales of different types of objects are different, and the spatial distribution is relatively complex. When extracting land use and land cover information from remote sensing images, convolutional neural networks use fixed convolution kernels to perform feature extraction using convolution operations. When the size of the convolution kernel does not match the scale of the object, it is impossible to accurately extract the details and global context information of objects of different scales, and it is difficult to capture edge feature information of objects of different scales. Convolutional neural networks need to extract features through multiple downsampling convolutions and pooling operations. More convolution and pooling operations will also cause the loss of edge detail feature information of some objects, further affecting the final accuracy of land use and land cover information extraction.

[0006] Therefore, based on the above problems, a remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network is proposed to achieve lower computing resources and higher efficiency, and better adapt to the extraction of land use and land cover information from remote sensing images. Summary of the invention

[0007] The present invention proposes a remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network, which is implemented by the following technical scheme: A remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network, characterized in that it includes the following operating steps:

[0008] Step 1: Use lightweight EffientNet V2 as the backbone network to extract remote sensing image features at five stages and five scales, providing more detailed ground object details and edge features;

[0009] Step 2, in the backbone network, the Edge Residual Convolution structure is used to extract the low-level features of the land use and land cover information of the remote sensing images for Stage 1 and Stage 2, and the Inverted Residual Convolution structure is used to extract the intermediate and high-level features of the land use and land cover information of the remote sensing images for Stage 3, Stage 4 and Stage 5;

[0010] Step 3: Add the ECA channel attention mechanism to the output features of Stage 1, Stage 2, and Stage 3 to obtain more accurate feature information;

[0011] Step 4: Densely connected atrous spatial pyramid pooling is formed by combining dense connection and atrous spatial pyramid pooling. Multi-scale context information is captured through atrous convolution and pooling operations of different scales. The atrous rate of convolution is adjusted to change the size of the receptive field, thereby obtaining feature information of different scales.

[0012] Step 5. A pyramid multi-scale unit module PMU is proposed to further extract low-level, medium-level and high-level feature information of different scales through 3×3 convolution, and then multi-scale feature extraction is performed through 4 3×3 hole-depth separable convolutions. Finally, the detailed information of land use and land cover is further captured through the fusion of features of different scales.

[0013] Preferably, in the Step 2, the original remote sensing image is input into the EdgeResidual Convolution as C×H×W, and the feature extracted by the 3×3 convolution is X=[X1,X2,X3...,X n ], and perform batch normalization operation.

[0014] Preferably, formula (1) is used to normalize each small batch of training data:

[0015]

[0016] Among them, γ and μ are learnable parameters, which perform offset and scaling operations respectively, and ε prevents the denominator from being 0;

[0017] Using formula (2), SiLU activation function, introduces stronger nonlinear expression ability and gradient information. The formula of SiLU function is:

[0018]

[0019] Where x is the input feature, and the input feature value is multiplied by the corresponding Sigmoid function value.

[0020] Preferably, Formula 3 is used in Step 3 to obtain more accurate feature information in the three channels by enhancing useful feature channels and weakening irrelevant feature channels. The ECA attention formula is as follows:

[0021]

[0022] where Y c is the input feature map X channel c attention, X c is the cth channel of the input feature map, H and W are the height and width of the feature map, σ is the Sigmoid activation function that normalizes the weights to between [0,1], and λ c,d is the learnable parameter in the convolution kernel of one-dimensional convolution, X d,i,j is the element value of the d-th channel position i,j in the input feature map.

[0023] Preferably, in the Step 4, the void ratio is designed to be a sawtooth structure, and a 3×3 void convolution is used. Through calculation of Formula 4 and repeated experiments, five void convolutions with void ratios of 2, 5, 9, 11, and 19 are used, and the formula is:

[0024] D i =max[D i+1 -2r i ,D i+1 -2(D i+1 -r i ),r i ] (4)

[0025] where Y i represents the value of position i of the output feature map of layer L, r L is the void ratio of the Lth layer, x(i+r L n) represents the position i+r of the Lth layer input feature map after the dilation rate L The value of n, w(λ) is the weight, and n is the position of all convolution kernels traversed;

[0026] In the Step 4, the feature graphs of each layer input are calculated by formula 5 to obtain the features of each layer after the 3×3 convolution operation with different void rates. The formula is:

[0027]

[0028] where Y i represents the value of position i of the output feature map of layer L, r L is the void ratio of the Lth layer, x(i+r L n) represents the position i+r of the Lth layer input feature map after the dilation rate L The value of n, w(λ) is the weight, and n is the position of all convolution kernels traversed;

[0029] Then, using the dense connection strategy, through formula 6, the output of the mth layer is merged through the first m-1 layers, and the output feature map is obtained through nonlinear changes. The formula is:

[0030] y m =σ m ([y0,y1,y2,···,y m-1 ]) (6)

[0031] where y m is the output feature map of the mth layer, [y0,y1,y2,···,y m-1 ] is the concatenation of the first m-1 feature maps, σ m is the linear transformation of the mth layer.

[0032] Preferably, in Step 5, the low-level feature X is input into the PMU. L (C2, H2, W2), intermediate feature X M (C1, H1, W1), advanced feature X H (C3, H3, W3), where the intermediate feature X L (C1,H1,W1) and advanced features X H (C3, H3, W3) uses formula 7 to restore the resolution after bilinear upsampling operation, the formula is:

[0033] D(m,n)=α·β·F(x1,y1)+(1-α)·β·F(x2,y1)+α·(1-β)·F(x1,y2)+(1-α)·(1-β)·F(x2,y2) (7)

[0034] Where D(m,n) is the pixel of the output feature map, F(m,n) is the value of the input feature map at position (m,n), x1, x2, y1, y2 are the coordinates of the nearest pixel, and α and β are the horizontal and vertical interpolation weights.

[0035] Preferably, formula (8) is used to first perform depthwise convolution on each input channel independently, and the formula is:

[0036] H d (m,n,c)=∑ j,k P c (m+r·j,n+r·k)·K c (j,k) (8)

[0037] Among them, H d (m,n,c) is the output of the hole depth convolution, (j,k) is the coordinate of the traversal convolution kernel, (m,n) is the specific position, c is the channel, r is the hole rate, K c is the convolution kernel passing through C, P c (m+r·j,n+r·k) is the position of the C channel input feature map with a hole rate of r relative to the position (m,n).

[0038] Preferably, formula (9) is used to merge the output channels of the depthwise convolution using 1×1 convolution, and after feature extraction through four hole-depth separable convolutions, all feature information is concatenated to obtain the final output feature map, which is:

[0039] H p (m,n)=∑ c H d (m,n,c)·ω(c) (9)

[0040] Among them, H p (m,n) is the output of the point convolution, and ω(c) is the weight of the Cth channel of the 1×1 convolution kernel.

[0041] Beneficial effects:

[0042] In view of the problems that deep convolutional neural networks (DCNNs) have high computational complexity, insufficient multi-scale feature extraction capabilities, and cannot accurately extract edge information of objects when extracting land use and land cover information from remote sensing images, the present invention proposes a lightweight multi-scale feature fusion network LMFF-Net. By adopting the improved EffcientNetV2 as the backbone network, LMFF-Net significantly reduces the demand for computing resources and enhances the ability to extract multi-scale features. By integrating the channel attention mechanism, LMFF-Net strengthens the recognition and attention to key features of the edge of objects. In addition, a new PMU module is developed to optimize the fusion effect of multi-scale features, and by improving the DenseAspp module, the ability to capture details of objects of different scales is further improved. The present invention deeply studies a new method of deep learning technology in the extraction of land use and land cover information from remote sensing images, and provides a new method reference for the extraction of land use and land cover information from remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the detailed structure diagram of LMFF-Net in the present invention;

[0044] Figure 2 It is the structure diagram of EdgeResidualConvolution in the present invention;

[0045] Figure 3 It is the structural diagram of Inverted Residual Convolution in the present invention;

[0046] Figure 4 This is the ECA attention mechanism diagram in the present invention;

[0047] Figure 5 It is a diagram of the module for improving densely connected void space pyramid pooling in the present invention;

[0048] Figure 6 This is the pyramid multi-scale unit PMU diagram in the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example:

[0051] like Figure 1-Figure 6As shown, the present invention proposes a remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network. The LMFF-Net network backbone adopts the lightweight EfficientNetV2 network proposed by Google researchers in 2021. The backbone network is mainly used for multi-scale feature extraction of land use and land cover information in remote sensing images. By using a large number of deeply separable convolutions in the backbone network, the number of parameters and the occupied computing resources of the convolutional neural network are reduced to improve the efficiency and computing speed of the model, and meet the needs of large-scale remote sensing image land use and land cover information extraction. By performing different scale feature extraction at 5 different stages of the backbone network, more refined ground object detail information and edge features are provided. In the backbone network, Stage1 and Stage2 are used as low-level features between different scales for extracting land use and land cover information from remote sensing images, and Stage3 is used as intermediate features between different scales for extracting land use and land cover information from remote sensing images. By adding the ECA channel attention mechanism to the features extracted in Stage 1, Stage 2 and Stage 3, the important information of feature extraction is enhanced, redundant information is suppressed, and the accuracy and generalization ability of land use and land cover information extraction of the backbone network are improved. Stage 3, Stage 4 and Stage 5 are intermediate and advanced features of different scales for extracting land use and land cover information from remote sensing images. The PMU module that can process ground object information and resolution of different scales is input to better extract ground object details and rich contextual information in remote sensing images, and improve the extraction accuracy of land use and land cover edge information. The improved dense atrous spatial pyramid pooling module DenseASPP is further used for advanced features. The receptive field is expanded through atrous convolution, and the multi-scale intra-features from fine-grained to coarse-grained ground object information are obtained from remote sensing images. The multi-scale intra-features are densely connected and fused to more finely capture the boundaries and details of land use and land cover information, and improve the model's understanding and learning ability of remote sensing image semantic information. Better modeling and extraction of abstract and high-level semantic features of land use and land cover information at different scales are carried out. Finally, through the effective fusion of low-level features, intermediate features and high-level features, the extraction results of land use and land cover information are obtained after upsampling, which effectively improves the training efficiency of the LMFF-Net network model and the accuracy of land use and land cover information extraction.

[0052] Specifically, it includes the improved EfficientNet V2 backbone network multi-scale feature extraction:

[0053] In order to extract multi-scale features of different sizes of objects in remote sensing images and improve the perception of objects of different scales and remote sensing image details, the lightweight EffientNet V2 is used as the backbone network to extract remote sensing image features of 5 scales in 5 stages. As shown in Table 1, the remote sensing image with an input size of 8×256×256 is extracted through the Stage1 and Stage2 stages in the backbone network. Figure 1 The low-level features of / 2 and 1 / 4 size are used to capture the edge contours, textures and other features of the objects in the remote sensing image, which can better preserve the details and location information of the object type. Figure 1 The mid-level features of the size of / 8 are more abstract and are mainly used for object type recognition and semantic information capture. Finally, high-level features of the size of 1 / 16 and 1 / 32 of the remote sensing image are extracted through Stage 4 and Stage 5 to obtain more abstract and semantically rich features, and to obtain the categories and spatial relationships between objects.

[0054] Table 1 EfficientNet V2 multi-scale feature extraction

[0055]

[0056] In the backbone network, the EdgeResidual Convolution structure is used to extract low-level features of land use and land cover information in remote sensing images in Stage 1 and Stage 2. Figure 2 This structure is mainly used to enhance the extraction of edge information of ground objects and refine the edges of different ground objects. It mainly extracts contours and edge features in remote sensing images through edge detection convolution, introduces nonlinear activation functions to enhance the expression ability of edge features, strengthens the extraction of edge information, and uses residual connections to add edge-enhanced feature maps to original image feature maps, effectively realizing feature fusion and information transmission.

[0057] The original remote sensing image input in EdgeResidual Convolution is C×H×W, and the features extracted by 3×3 convolution are X=[X1,X2,X3...,X n], perform Batch Normalization operation. Formula 1 is used to normalize each small batch of training data so that the output of the middle layer of the network has zero mean and unit variance, which accelerates the convergence of the network, reduces the vanishing or explosion of gradients, and maintains the overall stability of the training network. Formula 2, SiLU (Sigmoid Linear Unit) activation function, introduces stronger nonlinear expression ability and gradient information, which is conducive to enhancing the expression ability of edge features and the convergence of the algorithm. By connecting multiple 3×3 convolutions, Batch Normalization and SiLU nonlinear convolution blocks in series, more complex features in remote sensing images can be learned, which is more effective in extracting detailed information such as land use and land cover shape, edge features, etc. in remote sensing images.

[0058]

[0059] Among them, γ and μ are learnable parameters, which perform offset and scaling operations respectively, and ε is a very small parameter to prevent the denominator from being zero.

[0060] The formula of the SiLU function is:

[0061]

[0062] Where x is the input feature, and the input feature value is multiplied by the corresponding Sigmoid function value.

[0063] The Inverted Residual Convolution structure is used to extract the intermediate and advanced features of land use and land cover information from remote sensing images in Stage 3, Stage 4 and Stage 5. Figure 3 The input remote sensing image increases the dimension of the feature channel through point-by-point convolution (1×1 convolution) to extract richer and more complex features, adds Batch Normalization to reduce gradient vanishing or gradient explosion, enhances nonlinear features through the nonlinear activation function SiLU, and divides the convolution operation into two steps: depthwise convolution and point-by-point convolution through depthwise separable convolution.

[0064] In the original EfficientNet V2 backbone network, the context information in the extraction of Stage1 and Stage2 edge features and Stage2 mid-level features is insufficient, which easily makes it difficult to distinguish between mid- and low-level features and background, affecting the accuracy of edge feature information extraction. The ECA channel attention mechanism is added to the output features of Stage1, Stage2, and Stage3 to better focus on the main feature information, such as Figure 4, by performing global average pooling on the feature maps obtained by the above three channels in the spatial dimension, reducing the dimension and extracting the global information, Formula 3 is used to obtain more accurate feature information in the three channels by enhancing useful feature channels and weakening irrelevant feature channels.

[0065] The ECA attention formula is as follows:

[0066]

[0067] where Y c is the input feature map X channel c attention, X c is the cth channel of the input feature map, H and W are the height and width of the feature map, σ is the Sigmoid activation function that normalizes the weights to between [0,1], and λ c,d is the learnable parameter in the convolution kernel of one-dimensional convolution, X d,i,j is the element value of the d-th channel position i,j in the input feature map.

[0068] The present invention partially modifies the original EfficientNet V2 network. The modified EfficientNet V2 network consists of a series of repeated structures, and the specific structures and quantities are shown in Table 2. Through the combination of these structural blocks, feature information of different scales in remote sensing images can be effectively extracted. While improving the accuracy of LMFF-Net, the computational complexity and parameter requirements of the model are relatively reduced, which can effectively alleviate the problems of excessive consumption of computing resources and large number of parameters in the feature extraction process of deep convolutional neural networks.

[0069] Table 2 Detailed structure of the improved EfficientNet V2

[0070]

[0071] Specifically, densely connected void space pyramid pooling improvements:

[0072] Dense Connection Atrous Spatial Pyramid Pooling (DenseAspp) is a method that combines dense connection (DenseConnection) and atrous spatial pyramid pooling (ASPP). In dense connection, the features of each layer are spliced ​​with the features of all previous layers, which strengthens feature representation and gradient propagation, and helps to alleviate gradient disappearance. When using atrous spatial pyramid pooling, multi-scale context information is captured mainly through atrous convolution and pooling operations of different scales, and the size of the receptive field is changed by adjusting the atrous rate of the convolution, so as to obtain feature information of different scales.

[0073] After performing multiple dilated convolution operations, a large dilated ratio can easily lead to uneven sampling of the convolution kernel on the feature map. This will make the spatial structure of the output feature map more regular and grid-like, resulting in the loss of detail information, blurred edges, and the coarsening of the segmentation results, resulting in the grid effect. When the relationship between the feature map size and the dilated convolution dilated ratio is unreasonable, the grid effect is more obvious. In order to avoid and solve the grid effect generated in the extraction of land use and land cover information, in the original DenseAspp module, the present invention draws on the idea of ​​HDC to design the dilated coefficient of the convolution kernel so that it completely covers the low-level feature layer when extracting land use and land cover information, thereby improving the performance and accuracy of the dilated convolution. The dilated ratio is designed to be a sawtooth structure, and a 3×3 dilated convolution is used. Through the calculation of formula 4 and repeated experiments, 5 dilated convolutions with dilated ratios of 2, 5, 9, 11, and 19 are used to effectively extract multi-scale features of remote sensing image land use and land cover information. A smaller void ratio is used to extract short-distance information features, and a larger void ratio is used to extract long-distance information features. Figure 5 As shown, the input feature map of each layer is calculated by formula 5 to obtain the features of each layer after 3×3 convolution operation with different void ratios. Then, the dense connection strategy is adopted, and the output of the mth layer is merged through the first m-1 layers through formula 6, and the output feature map is obtained through nonlinear changes. The improved DenseAspp module of the present invention adopts a smaller void ratio, which can enhance the perception of land use and land cover information, thereby improving the ability of LMFF-Net to extract land use and land cover information.

[0074] D i =max[D i+1 -2r i ,D i+1 -2(D i+1 -r i ),r i ] (4)

[0075] Where D i refers to the maximum void ratio in the i-th layer, r i is the dilation rate of the i-th layer. If the kernel is an n×n convolution, D2≤n.

[0076]

[0077] where Y i represents the value of position i of the output feature map of layer L, r L is the void ratio of the Lth layer, x(i+r L n) represents the position i+r of the Lth layer input feature map after the dilation rate LThe value of n, w(λ) is the weight, and n is the position of all convolution kernels traversed.

[0078] y m =σ m ([y0,y1,y2,···,y m-1 ]) (6)

[0079] where y m is the output feature map of the mth layer, [y0,y1,y2,···,y m-1 ] is the concatenation of the first m-1 feature maps, σ m is the linear transformation of the mth layer.

[0080] Specifically, the pyramid multi-scale unit PMU:

[0081] The traditional spatial pyramid method has a fixed scale and cannot effectively extract objects of different sizes. When fusing multi-scale features, the results are spliced ​​after independent pooling, resulting in no correlation between scales. At the same time, the shortcomings of ordinary convolution such as limited receptive field and large number of parameters are avoided. The present invention proposes a pyramid multi-scale unit module PMU (Pyramid Multi-scale Unit). Figure 6 , PMU further extracts low-level, medium-level, and high-level feature information of different scales through 3×3 convolution, better adapts to the feature extraction of targets of different sizes, and improves the consistency of feature expression by fusing the context information of multi-scale features. The bilinear interpolation method is used to upsample and translate the middle and deep features, and the transition layer composed of nonlinear transformation mapping is introduced to improve the convergence speed and reasoning accuracy of training. The fused feature information is then extracted through four 3×3 atrous depthwise separable convolutions (Atrous Depthwise Separable Convolution) for multi-scale feature extraction. 3×3 atrous depthwise separable convolutions with atrous rates of 1, 2, 5, and 9 are used for multi-scale feature extraction. The jagged nature of such a structure can simultaneously meet the information extraction of smaller land use and land cover and larger land use and land cover. By expanding the receptive field range, a wider range of contextual information is obtained, and fine features of land use and land cover spatial information are extracted. At the same time, the use of depthwise separable convolution can effectively reduce the number of parameters and calculations, achieving the purpose of lightweight design. Finally, by integrating features of different scales, detailed information on land use and land cover can be further captured.

[0082] Input low-level feature X into the PMU module L (C2, H2, W2), intermediate feature X M (C1, H1, W1), advanced feature X H (C3, H3, W3), where the intermediate feature XL (C1,H1,W1) and advanced features X H (C3, H3, W3) is subjected to bilinear upsampling using Formula 7 to restore the resolution. After concatenation, a feature map X(C1+C2+C3, H2, W2) is obtained. Formula 8 is used to first perform depthwise convolution independently on each input channel, and then Formula 9 is used to merge the output channels of the depthwise convolution using 1×1 convolution. After feature extraction through four hole-depth separable convolutions, all feature information is concatenated to obtain the final output feature map.

[0083] D(m,n)=α·β·F(x1,y1)+(1-α)·β·F(x2,y1)+α·(1-β)·F(x1,y2)+(1-α)·(1-β)·F(x2,y2) (7)

[0084] Where D(m,n) is the pixel of the output feature map, F(m,n) is the value of the input feature map at position (m,n), x1, x2, y1, y2 are the coordinates of the nearest pixel, and α and β are the horizontal and vertical interpolation weights.

[0085] H d (m,n,c)=∑ j,k P c (m+r·j,n+r·k)·K c (j,k) (8)

[0086] Among them, H d (m,n,c) is the output of the hole depth convolution, (j,k) is the coordinate of the traversal convolution kernel, (m,n) is the specific position, c is the channel, r is the hole rate, K c is the convolution kernel passing through C, P c (m+r·j,n+r·k) is the position of the C channel input feature map with a hole rate of r relative to the position (m,n).

[0087] H p (m,n)=∑ c H d (m,n,c)·ω(c) (9)

[0088] Among them, H p (m,n) is the output of the point convolution, and ω(c) is the weight of the Cth channel of the 1×1 convolution kernel.

[0089] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network, characterized by: The steps are as follows: Step 1: Use lightweight EffientNet V2 as the backbone network to extract remote sensing image features at five stages and five scales, providing detailed information and edge features of ground objects; Step 2, in the backbone network, the Edge Residual Convolution structure is used to extract the low-level features of the land use and land cover information of the remote sensing images for Stage 1 and Stage 2, and the Inverted Residual Convolution structure is used to extract the intermediate and high-level features of the land use and land cover information of the remote sensing images for Stage 3, Stage 4 and Stage 5; Step 3, add the ECA channel attention mechanism to the output features of Stage 1, Stage 2, and Stage 3 to obtain feature information; Step 4: Densely connected atrous spatial pyramid pooling is formed by combining dense connection and atrous spatial pyramid pooling. Multi-scale context information is captured through atrous convolution and pooling operations of different scales. The atrous rate of convolution is adjusted to change the size of the receptive field, thereby obtaining feature information of different scales. Step 5. A pyramid multi-scale unit module PMU is proposed to further extract low-level, medium-level and high-level feature information of different scales through 3×3 convolution, and then multi-scale feature extraction is performed through 4 3×3 hole-depth separable convolutions. Finally, the detailed information of land use and land cover is further captured through the fusion of features of different scales.

2. The remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network according to claim 1, characterized in that: In the Step 2, the original remote sensing image is input into the Edge Residual Convolution as C×H×W, and the feature extracted by the 3×3 convolution is X=[X1,X2,X3...,X n ], and perform batch normalization operation.

3. The remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network according to claim 2, characterized in that: Formula (1) is used to normalize each small batch of training data: Among them, γ and μ are learnable parameters, which perform offset and scaling operations respectively, and ε prevents the denominator from being 0; Using formula (2), SiLU activation function, introduces stronger nonlinear expression ability and gradient information. The formula of SiLU function is: Where x is the input feature, and the input feature value is multiplied by the corresponding Sigmoid function value.

4. The remote sensing image semantic segmentation method based on a lightweight multi-scale convolutional neural network according to claim 1, characterized in that: Formula 3 is used in Step 3 to enhance useful feature channels and weaken irrelevant feature channels to obtain more accurate feature information in the three channels. The ECA attention formula is as follows: where Y c is the input feature map X channel c attention, X c is the cth channel of the input feature map, H and W are the height and width of the feature map, σ is the Sigmoid activation function that normalizes the weights to between [0,1], and λ c,d is the learnable parameter in the convolution kernel of one-dimensional convolution, X d,i,j is the element value of the d-th channel position i,j in the input feature map.

5. The remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network according to claim 1, characterized in that: In the Step 4, the dilation rate is designed into a sawtooth structure, and a 3×3 dilated convolution is used. Through the calculation of Formula 4 and repeated experiments, five dilated convolutions with dilation rates of 2, 5, 9, 11, and 19 are used, and the formula is: D i =max[D i+1 -2r i ,D i+1 -2(D i+1 -r i ),r i ] (4) where Y i represents the value of position i of the output feature map of layer L, r L is the void ratio of the Lth layer, x(i+r L n) represents the position i+r of the Lth layer input feature map after the dilation rate L The value of n, w(λ) is the weight, and n is the position of all convolution kernels traversed; In the Step 4, the feature graphs of each layer input are calculated by formula 5 to obtain the features of each layer after the 3×3 convolution operation with different void rates. The formula is: where Y i represents the value of position i of the output feature map of layer L, r L is the void ratio of the Lth layer, x(i+r L n) represents the position i+r of the Lth layer input feature map after the dilation rate L The value of n, w(λ) is the weight, and n is the position of all convolution kernels traversed; Then, using the dense connection strategy, through formula 6, the output of the mth layer is merged through the first m-1 layers, and the output feature map is obtained through nonlinear changes. The formula is: y m =σ m ([y0,y1,y2,···,y m-1 ]) (6) where y m is the output feature map of the mth layer, [y0,y1,y2,···,y m-1 ] is the concatenation of the first m-1 feature maps, σ m is the linear transformation of the mth layer.

6. The remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network according to claim 1, characterized in that: In Step 5, the low-level feature X is input into the PMU. L (C2, H2, W2), intermediate feature X M (C1,H1,W1), advanced features X H (C3, H3, W3), where the intermediate feature X L (C1, H1, W1) and advanced features X H (C3, H3, W3) uses formula 7 to restore the resolution after bilinear upsampling operation, the formula is: D(m,n)=α·β·F(x1,y1)+(1-α)·β·F(x2,y1)+α·(1-β)·F(x1,y2)+(1-α)·(1-β)·F(x2,y2) (7) Where D(m,n) is the pixel of the output feature map, F(m,n) is the value of the input feature map at position (m,n), x1, x2, y1, y2 are the coordinates of the nearest pixel, and α and β are the horizontal and vertical interpolation weights.

7. The remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network according to claim 6, characterized in that: Using formula (8), we first perform depthwise convolution on each input channel independently, as follows: H d (m,n,c)=∑ j,k P c (m+r·j,n+r·k)·K c (j,k) (8) Among them, H d (m,n,c) is the output of the hole depth convolution, (j,k) is the coordinate of the traversal convolution kernel, (m,n) is the specific position, c is the channel, r is the hole rate, K c is the convolution kernel passing through C, P c (m+r·j,n+r·k) is the position of the C channel input feature map with a hole rate of r relative to the position (m,n).

8. The remote sensing image semantic segmentation method based on lightweight multi-scale convolutional neural network according to claim 7, characterized in that: Formula (9) is used to merge the output channels of the depthwise convolution using 1×1 convolution. After feature extraction through four hole-depth separable convolutions, all feature information is concatenated to obtain the final output feature map. The formula is: H p (m,n)=∑ c H d (m,n,c)·ω(c) (9) Among them, H p (m,n) is the output of the point convolution, and ω(c) is the weight of the Cth channel of the 1×1 convolution kernel.

Images