Landslide image segmentation method based on high and low semantic information fusion

Abstract

The present invention discloses a landslide image segmentation method based on high-low semantic information fusion, and belongs to the technical field of landslide image processing. The present invention constructs a semantic segmentation network for landslide remote sensing images based on high-low semantic information fusion, and obtains multi-scale information of landslide images by constructing an enhanced global information pooling module in the network; constructs different feature interaction modules between the encoder and the decoder to improve the information interaction ability of the model, and pays attention to the characteristics of the target area of ​​the image by enhancing the target information attention module; designs a multi-level semantic information fusion module to refine the boundary information of the image, improve the model's recognition ability of the landslide area, and improve the segmentation effect of the model.

Description

Technical Field

[0001] The invention relates to a landslide image segmentation method based on high and low semantic information fusion, and belongs to the technical field of landslide image processing. Background Art

[0002] Landslide is one of the most frequent natural disasters in the world. Generally, under the action of external forces, the surface soil, rocks and other materials will move along a certain sliding surface, causing changes in the terrain. Landslide disasters can cause casualties, property losses and damage to the ecological environment. Therefore, efficient and accurate identification of landslide areas is of great significance for preventing geological disasters, protecting the ecological environment and human property safety.

[0003] In the early days, landslide surveys were mainly conducted by geological experts to collect landslide data on site, which consumed a lot of time and had certain safety hazards. With the development of remote sensing technology, researchers can quickly obtain the distribution of landslide areas after disasters by collecting relevant images of the study area through advanced high-resolution technical equipment such as satellites. For example, WANG et al. used InSAR technology to monitor landslides around the reservoir before and after water storage, and verified the effectiveness of InSAR in landslide identification; YANG et al. used multimodal remote sensing data including digital elevation models and optical remote sensing to identify and predict landslide areas in mountainous areas of Yunnan. However, landslides often occur in mountains, and a large number of vegetation and trees will block the landslide area. At the same time, the color characteristics of newly occurring landslide areas are mostly light tones such as gray and blue, while the landslide areas in remote sensing images of old landslide areas are often different in color due to the recovery of surface vegetation, making them difficult to distinguish. Therefore, how to accurately identify landslide areas is one of the difficulties in solving the identification and surveying problems using remote sensing images.

[0004] Domestic and foreign researchers have proposed many methods in the field of semantic segmentation of landslide images. LI et al. introduced the MobileNetv2 structure into the PSPNet model to reduce the number of network parameters and computational complexity, and reduce landslide classification errors. YANG et al. used ViT in ResU-Net and introduced CBAM to improve the segmentation effect of the model for landslide images. GARCIA et al. proposed a semi-automatic CNN model for identifying landslides in rainforest areas, which can identify landslide areas in remote sensing images, but the model loses its detailed features. NIU et al. proposed a landslide segmentation model called Reg-SA-Unet++. By simplifying the structure of Unet++ and introducing an attention module at the location where the convolutional block is connected, the parameters are reduced while improving the model's feature extraction ability. However, the small amount of data and the artificially labeled areas will affect the segmentation effect of the model, and there are certain errors. CHEN et al. proposed a new type of convolutional cross-dual network to identify landslide data information collected by satellites, combining the advantages of ConvNeXt and Transformer to enhance the model's ability to extract spatial information. Based on the DeepLabv3+ model, ZHENG et al. proposed a landslide segmentation model that integrates terrain features and embedded multi-channels. They used geological knowledge and deep learning to identify landslides caused by the Jiuzhaigou earthquake, effectively improving the recognition effect of landslide areas in complex backgrounds. However, in the irregular landslide areas caused by the earthquake, the corresponding shape features of the area were not highlighted. LV et al. proposed a segmentation model based on pyramid visual transformers to identify landslides of different shapes. Based on the self-attention module and boundary attention operation, the model extracts multi-scale features while enhancing the model's capture of shape information at each stage, which well illustrates the advantages of the model in the segmentation effect of landslide remote sensing images.

[0005] However, in the actual detection process, due to the influence of factors such as climate and mountain terrain, the size of landslides has no qualitative standard, and most of them are irregular in shape. How to maintain the overall shape of the area is a problem that the segmentation model needs to consider. Secondly, highlighting the boundary information of the landslide area is also a difficulty in landslide image recognition. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a landslide image segmentation method based on high and low semantic information fusion, improve the model's recognition ability of the landslide area by refining edge information, and enhance the segmentation effect of the model.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The landslide image segmentation method based on high and low semantic information fusion includes the following steps:

[0009] Step 1, obtaining a landslide remote sensing image dataset and preprocessing the dataset, dividing the preprocessed dataset into a training set, a validation set, and a test set;

[0010] Step 2, building a semantic segmentation network for landslide remote sensing images based on high and low semantic information fusion, including an encoder and a decoder, wherein the encoder includes a backbone network, an enhanced global information pooling module, and an enhanced target information attention module, and the decoder includes a first to second feature interaction module and a first to second multi-level semantic information fusion module;

[0011] After the input image of the encoder passes through the backbone network, the backbone network outputs low-level landslide feature maps with image sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image respectively. The low-level landslide feature map with an image size of 1 / 32 of the input image passes through the enhanced global information pooling module and the enhanced target information attention module in turn to obtain a high-level landslide feature map. ;

[0012] The low-level landslide feature maps with image sizes of 1 / 8 and 1 / 16 of the input image are stacked together to generate feature maps through the first feature interaction module. , After upsampling, Overlay output feature map , and Input into the first multi-level semantic information fusion module and output feature map The low-level landslide feature maps with image sizes of 1 / 4 and 1 / 8 of the input image are stacked together to generate feature maps through the second feature interaction module. , After upsampling, Overlay output feature map , and Input into the second multi-level semantic information fusion module and output feature map ; The low-level landslide feature map with an image size of 1 / 4 of the input image is convolved by 1×1 to generate a feature map , and After fusion, 1×1 convolution and LeakyReLU activation function are performed in sequence to generate feature maps , and After multiplication, the decoder output image is obtained after 3×3 convolution and upsampling operations.

[0013] Step 3, input the training set and the validation set into the landslide remote sensing image semantic segmentation network based on high and low semantic information fusion for training and validation, calculate the loss function and perform back propagation, update the network parameters, and obtain the trained optimal parameter network;

[0014] Step 4: Input the test set into the trained optimal parameter network and output the segmentation effect diagram of the landslide image.

[0015] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0016] Obtain a landslide remote sensing image dataset, including landslide remote sensing images, label maps corresponding to landslide remote sensing images, and non-landslide remote sensing images;

[0017] Perform data enhancement operations on the landslide remote sensing images in the dataset, including mirroring and flipping;

[0018] The data set after data augmentation operation is divided into training set, validation set and test set according to preset proportions.

[0019] As a preferred solution of the present invention, the backbone network includes an expanded convolution layer, a depth-separable convolution layer, an SE attention module and a first convolution layer. The input of the backbone network is sequentially passed through the expanded convolution layer and the depth-separable convolution layer to obtain a first output. The first output is passed through the SE attention module to obtain a second output. The first output is multiplied by the second output and passed through the first convolution layer to obtain a third output. The third output is multiplied by the input of the backbone network to obtain the output of the backbone network. The convolution kernel size of the expanded convolution layer is 1×1, the convolution kernel size of the depth-separable convolution layer is 3×3, and the convolution kernel size of the first convolution layer is 1×1.

[0020] The depthwise separable convolution layer includes a depthwise convolution layer and a pointwise convolution layer connected in sequence, and the depthwise convolution layer and the pointwise convolution layer are both connected with batch normalization and ReLU activation functions; the convolution kernel size of the pointwise convolution layer is 1×1×M, where M is the input dimension;

[0021] The SE attention module includes a global average pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence, and the first fully connected layer is connected with a ReLU activation function, and the second fully connected layer is connected with a hwish activation function.

[0022] As a preferred embodiment of the present invention, the enhanced global information pooling module includes second to fourth convolutional layers, first to sixth dilated convolutional layers, global average pooling layers and strip pooling layers, the second to fourth convolutional layers are connected with batch normalization and ReLU activation functions, the first to sixth dilated convolutional layers are connected with batch normalization and ReLU activation functions, the convolution kernel sizes of the second to fourth convolutional layers are 1×1, the convolution kernel sizes of the first to sixth dilated convolutional layers are 3×3, the expansion rates of the first to third dilated convolutional layers are 1, 2 and 3 respectively, and the expansion rates of the fourth to sixth dilated convolutional layers are 1, 2 and 5 respectively;

[0023] The backbone network outputs a low-level landslide feature map with an image size of 1 / 32 of the input image. As the input of the enhanced global information pooling module, the feature map is output through the second convolutional layer , The feature map is generated by the third convolutional layer , The feature map is generated by the first, second and third hole convolution layers in sequence , The feature map is generated by the fourth, fifth and sixth hole convolution layers in sequence , Generate feature map through global average pooling layer , Generate feature map through strip pooling layer ,Will , , , and After cascading, Fusion: The fusion result is passed through the fourth convolutional layer to obtain the output of the enhanced global information pooling module. .

[0024] As a preferred solution of the present invention, the structures of the first to second feature interaction modules are the same, both of which include the fifth to sixth convolutional layers, an enhanced target information attention module and a ReLU activation function, and the fifth to sixth convolutional layers are both connected with batch normalization, the convolution kernel size of the fifth convolutional layer is 1×1, and the convolution kernel size of the sixth convolutional layer is 3×3;

[0025] For the two inputs of each feature interaction module, the input with the smallest image size After upsampling, the sixth convolutional layer is used to obtain the feature map ; Enter another The feature map is obtained through the fifth convolutional layer ,Will and The result of fusion As the input of the enhanced target information attention module, the output of the enhanced target information attention module is activated by the ReLU function to obtain the output of each feature interaction module.

[0026] As a preferred solution of the present invention, the enhanced target information attention module includes the seventh to eighth convolutional layers, the first to second channel attention units and the spatial attention unit, the seventh to eighth convolutional layers are connected with batch normalization and LeakyReLU activation functions, and the convolution kernel sizes of the seventh to eighth convolutional layers are both 1×1;

[0027] Enhance the input of the attention module with target information There are three paths. After passing through the first channel attention unit and the seventh convolutional layer, the feature map is obtained. ; In the second path After the second channel attention unit and Multiply, and the result of the multiplication is passed through the spatial attention unit and Superposition, to obtain the feature map with increased target information attention ; In the third path After the eighth convolutional layer, the output of the enhanced target information attention module is obtained.

[0028] As a preferred solution of the present invention, the structures of the first to second multi-level semantic information fusion modules are the same, and both include the first to second average pooling layers, the ninth to eighteenth convolutional layers, the first to second average pooling layers are connected with convolutional layers and batch normalization, the ninth to eighteenth convolutional layers are connected with batch normalization and ReLU activation function, the convolution kernel sizes of the tenth, thirteenth, fourteenth, fifteenth, sixteenth and eighteenth convolutional layers are all 1×1, the convolution kernel sizes of the eleventh and seventeenth convolutional layers are all 3×3, the convolution kernel size of the ninth convolutional layer is 1×3, the convolution kernel size of the twelfth convolutional layer is 3×1, and the convolution kernel sizes of the convolutional layers connected after the first to second average pooling layers are all 1×1;

[0029] For the two inputs of each multi-level semantic information fusion module, the input with the smallest image size is Upsampling is performed, and the upsampling results are successively passed through the first average pooling layer and the LeakyReLU activation function to obtain the feature map ; The upsampling result is passed through the ninth convolutional layer to obtain the fourth output, and the fourth output is passed through the tenth, eleventh and twelfth convolutional layers respectively. The output results of the tenth, eleventh and twelfth convolutional layers are superimposed and then passed through the thirteenth convolutional layer to obtain the feature map ; and After superposition, we get the feature map ; Enter another After the fourteenth, fifteenth and sixteenth convolutional layers, the feature map is obtained. ; Another input The feature map is obtained by the second average pooling layer and the LeakyReLU activation function. ; and After superposition, we get the feature map ; and After fusion, the fusion results are passed through the seventeenth and eighteenth convolutional layers in turn to obtain the output of each multi-level semantic information fusion module.

[0030] As a preferred solution of the present invention, in step 3, the loss function is obtained by adding the Diess loss function and the cross entropy loss function, and the formula is as follows:

[0031] ,

[0032] in, and Denote the Diess loss function and the cross entropy loss function respectively, N Indicates the total number of target pixels. and Respectively represent the training network for the target pixel i The label value and predicted value of .

[0033] A computer device comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the landslide image segmentation method based on high-low semantic information fusion are implemented.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the landslide image segmentation method based on high-low semantic information fusion are implemented.

[0035] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0036] 1. The present invention constructs an enhanced global information pooling module to obtain global multi-scale information and improve the segmentation continuity of the landslide area.

[0037] 2. The present invention constructs different feature interaction modules between the encoder and the decoder to improve the information interaction ability of the model, reduce the loss of model area information, and improve the interaction of upper and lower semantic features. And the target information attention module is used to focus on the target area features of the image.

[0038] 3. The present invention designs a multi-level semantic information fusion module, which superimposes high-level semantic information with low-level feature information to achieve the fusion of different feature information and further refine the boundary information of the landslide area. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the landslide image segmentation method based on high and low semantic information fusion of the present invention;

[0040] Figure 2 It is a network structure diagram of semantic segmentation of landslide remote sensing images based on high and low semantic information fusion constructed by the present invention;

[0041] Figure 3 It is a structural diagram of the backbone network of the present invention;

[0042] Figure 4 It is a structural diagram of the enhanced global information pooling module of the present invention;

[0043] Figure 5 is a structural diagram of the feature interaction module of the present invention;

[0044] Figure 6 It is a structural diagram of the multi-level semantic information fusion module of the present invention;

[0045] Figure 7 It is a comparison chart of experimental indicators between the network proposed by the present invention and other networks;

[0046] Figure 8 It is a schematic diagram comparing the segmentation results of the network proposed in the present invention and other networks, wherein (a) is the original image, (b) is the label image, (c) is the SegNet network, (d) is the PSPNet network, (e) is the DA-Net network, (f) is the Deeplabv3+ network, and (g) is the network of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0048] like Figure 1 As shown, the present invention proposes a landslide image segmentation method based on high and low semantic information fusion, comprising the following steps:

[0049] Step 1: Obtain the Bijie landslide image dataset and the Sichuan landslide image dataset, preprocess the dataset and divide it into training set, validation set and test set according to the set ratio.

[0050] The acquired data sets are: Bijie landslide image data set and Sichuan landslide image data set, which include landslide images, label images corresponding to landslide images, and non-landslide images;

[0051] The data set preprocessing is as follows: data enhancement is performed on the original images in the Bijie landslide image dataset and the Sichuan landslide image dataset. The enhancement operations include mirroring and flipping to avoid overfitting of the model.

[0052] The data set is divided specifically as follows: the enhanced Bijie landslide image data set and the Sichuan landslide image data set are randomly divided into three parts, of which the training set and the validation set account for 80%, and the test set accounts for 20%.

[0053] Step 2: Build a landslide image semantic segmentation network based on the fusion of high and low semantic information, such as Figure 2 As shown in the figure, it mainly includes the backbone network Mobilenetv3, enhanced global information pooling module (EGIPM), enhanced target information attention module (ETIAM), feature interaction module (FIM) and multi-level semantic information fusion module (MSIFM). The backbone network and enhanced global information pooling module constitute the encoder (Encoder) of the landslide image semantic segmentation network, and the feature interaction module and multi-level semantic information fusion module constitute the decoder (Decoder).

[0054] Step 2.1: The data-enhanced landslide image dataset is first input into the backbone network to output high-level features The backbone network consists of two parts: convolution (conv), batch normalization (BN), activation function h-swish, depthwise separable convolution (DSConv) and average pooling layer (Pooling). NL indicates the type of activation function and FC indicates the fully connected layer. The image input to the backbone network first passes through the expansion layer of 1×1 convolution to increase the number of channels, and batch normalization and activation function are used to improve accuracy. Afterwards, two independent convolutions, depthwise convolution and pointwise convolution, are used to form overlapping depthwise separable volumes. At the same time, each layer of convolution is followed by BN and ReLU activation functions. In terms of convolution kernel selection, the depthwise convolution part uses a 3×3 convolution kernel, and a 5×5 convolution kernel is also used to efficiently focus on the channel dimension of the input image. The point-by-point convolution is a standard convolution with a convolution kernel size of 1×1×M (M is the input dimension). Finally, the channels of the input image are compressed through the average pooling layer, and the channels are adjusted through 1×1 convolution and combined with the input feature map to form a residual structure, so that the image is output from the backbone network.

[0055] Step 2.2: The feature image output in step 2.1 is passed through the enhanced global information pooling module, which mainly consists of convolution (Conv), global pooling (Global Pooling) and strip pooling (Strip Pooling). R represents the expansion rate. First, the input feature image Change the channel size of the feature image through 1×1 convolution and output the feature , It goes through five parts: 1×1 convolution, two groups of 3×3 convolutions with different dilation rates, strip pooling, and global pooling. Among them, the two groups of serially arranged 3×3 convolutions use variable dilation rates to enhance the receptive field, and the dilation of the dilated convolution is set to (1, 2, 3) and (1, 2, 5) respectively. BN and ReLU activation functions are also used in the 3×3 dilated convolution of each branch to reduce feature loss in the process and better obtain global multi-scale information through continuous convolution. The feature image can be obtained through these five parts. , , , as well as , and then output from the trunk Cascading with five features can effectively integrate the information of the feature map, and obtain the output feature image through 1×1 convolution .

[0056] Step 2.3: Pass the feature image outputted in step 2.2 through the enhanced target information attention module to better focus on the fine features of the landslide image and improve the network effect. The enhanced target information attention module uses the feature map outputted by the enhanced global information pooling module in step 2.2 as the input of the enhanced target information attention module and inputs it into the attention module. It will be divided into three paths. The first and second paths will first obtain channel information through CA, and then the first path will use 1×1 convolution and BN to obtain features. , the second path uses channel attention and feature Multiply and combine the feature image with the spatial attention image by summing the image pixel information to generate the enhanced target attention image feature Finally, the image channel size is changed by a 1×1 convolution with BN and LeakyReLU activation functions. The LeakyReLU activation function can enhance the weighting and screening of feature images, play a role of balancing optimization in the feature interaction process, improve the segmentation model's attention to the target features, and avoid global attention weighting of all feature maps.

[0057] Step 2.4: The landslide feature map output by the feature interaction module in step 2.3 and the landslide feature map after the low-level information output by the backbone network in step 2.1 are fused through the decoding module to obtain the final landslide remote sensing segmentation map. The decoder part mainly constructs the feature interaction module and the multi-level semantic information fusion module. Among them, L represents the LeakyReLU activation function, and UPsample represents the upsampling operation. First, the feature information output by step 2.3 in the low-level feature information is sent to the decoder, which will generate high-level semantic feature information. At the same time, the low-level features with an image size of 1 / 8 and the low-level feature information with an image size of 1 / 16 output by the backbone network are stacked together through the feature interaction module to generate , and then use the high-level semantic feature information of the upsampling operation Superimpose output feature image , feature image Further information about features Input into the multi-level semantic information fusion module and output feature image Similarly, the low-level features with a size of 1 / 4 of the image output by the backbone network and the low-level feature information with a size of 1 / 8 of the image are stacked together through the feature interaction module to generate , and then with the feature information Superimpose output feature image , feature image Further information about features Input into the multi-level semantic information fusion module and output feature image Finally, the feature image is output The low-level feature information with image size of 1 / 4 is convolved by 1×1 and then convolved with the output feature image again. The final landslide segmentation image is output through stacking, and the LeakyReLU activation function is used through 3×3 convolution and upsampling operations.

[0058] The structure of the backbone network is as follows Figure 3 As shown in the figure, the backbone network is mainly composed of batch normalization, activation function, depthwise separable convolution and average pooling layer; the role of batch normalization BN is to improve detection efficiency; activation functions include ReLU and h-swish, which can improve model accuracy; depthwise separable convolution consists of two independent convolutions: depthwise convolution and pointwise convolution; the average pooling layer performs global average pooling in the channel direction.

[0059] The structure diagram of the enhanced global information pooling module is as follows Figure 4 As shown in the figure, it is mainly composed of convolutional layers and pooling layers; the convolutional layers include: 1×1 convolution and 3×3 convolution with BN layer and ReLU activation function for feature extraction; the pooling layers include: global average pooling layer and strip pooling layer (SP), and the pooling layer is used to obtain multi-scale features. The strip pooling layer is a pooling operation method used for image processing. In traditional pooling operations, a fixed-size window is usually slid on the image, and then the maximum value or average value of the pixels in the window is taken as the pooling result. SP divides the image into multiple strip areas and performs pooling operations on the pixels in each strip area, which can provide more image information for the model and better preserve the structural information and texture features of the image.

[0060] The structure of the enhanced target information attention module is as follows Figure 5 As shown in the figure, it is mainly composed of modules such as spatial attention SA (SpatialAttention), channel attention CA (Channel Attention), and convolution.

[0061] The structure of the feature interaction module is as follows: Figure 5 As shown: It is mainly composed of convolutional layers, upsampling layers, weighted operations and splicing operations; the convolutional layer is composed of 1×1 convolution and 3×3 convolution; the upsampling layer performs upsampling operations through upsampling; the weighted operation multiplies the global context vector with the corresponding feature map; the splicing operation stacks the corresponding elements of different feature maps.

[0062] The feature interaction module will pass through 1×1 convolution and BN layer; the other part will perform upsampling operation, and then use 3×3 convolution to expand the receptive field of the feature image, and interact with it through the feature image generated by the BN layer, and then enhance the target area information through ETIAM, and finally combine with the ReLU function to output the final image features.

[0063] The structure of the multi-level semantic information fusion module is as follows: Figure 6 As shown in the figure, it is mainly composed of convolutional layers, upsampling layers, weighted operations and splicing operations; the convolutional layers are mainly composed of 1×1 convolutions and 3×3 convolutions; the upsampling layers perform upsampling operations through upsampling; the weighted operations multiply the global context vector with the corresponding feature maps; the splicing operations stack the corresponding elements of different feature maps.

[0064] Multi-level semantic information fusion modules share high-level semantic features and low-level semantic features Two inputs, The up-convolution operation is first used to keep the feature dimensions consistent, and then it passes through two different branches. The 1×1 convolution with BN and ReLU activation functions, AvgPool and LeakyReLU activation functions constitute the up-convolution. Will get features from this branch , keeping the number of channels unchanged.

[0065] The lower branch is composed of 1×1 convolution and three parallel convolutions: 3×3, 1×3, and 3×1. After 1×1 convolution, the feature , 1×3 convolution and 3×1 convolution are both one-dimensional convolution operations, which perform convolution along the horizontal and vertical directions of the input tensor respectively. 3×3 convolution is used to capture the local features of the image, and three features can be obtained , and The parallel design of three convolutions can better extract image feature information and improve the segmentation effect of the model. Get the feature through this branch .

[0066] On the other hand, low-level semantic features It will also go through two different branches. The lower branch is also composed of AvgPool, LeakyReLU activation function, and 1×1 convolution with BN and ReLU activation functions to obtain the feature map , and the upper branch is composed of three 3×3 convolutions with BN and ReLU activation functions. Through this branch, the module can enhance the acquisition of image detail information and keep the number of image channels unchanged, so that the feature map can be obtained. Similarly, the above features and Add to get the feature map The two output feature maps are cascaded, and then 3×3 convolution and 1×1 convolution operations are used to output the final feature map.

[0067] Step 3: Input the landslide images of the training set and validation set preprocessed in step 1 into the landslide image semantic segmentation network in step 2 for training and validation, calculate the loss function and perform back propagation, update the network parameters, and obtain the optimal parameter model.

[0068] Step 3.1: Randomly initialize the parameters of the landslide image semantic segmentation network, input the preprocessed training set and validation set data in step 1 into the landslide image segmentation network based on multi-layer feature information fusion in step 2, generate the semantic segmentation effect map of the landslide remote sensing image, and calculate the loss;

[0069] Step 3.2: Loss back propagation, update network parameters, minimize the loss function as the optimization goal, obtain the optimal parameter model and save it.

[0070] The loss function combines the Dice Loss function and the Cross-Entropy Loss (CE Loss) function. In actual detection, Dice Loss may experience loss saturation, while combining it with CE Loss can make the loss function decrease smoothly during training and improve the accuracy of the model for landslide image detection. The loss function is shown as follows:

[0071] ,

[0072] In the formula, and Respectively represent the model for the target i The label value and predicted value of N Indicates the total number of target pixels. In step 3.1, the training network is input with the Bijie landslide image, Sichuan landslide image and label image in step 1 for training the network. During the training of the Bijie landslide dataset, the accuracy of the network model on the test set is obtained. Accuracy , category recall MRecall and the average interaction ratio MIoU , the accuracy, category average accuracy and average interaction ratio formulas are as follows:

[0073] ,

[0074] In the formula, k represents the total number of categories of landslide images, Indicates the number of pixels that correctly identify the landslide area, Indicates the number of pixels that identify the landslide area as the background area, Indicates the number of pixels that identify the background area as a landslide area.

[0075] Step 4: Input the preprocessed test set in step 1 into the optimal parameter model trained in step 3, and output the segmentation effect diagram of the landslide remote sensing image.

[0076] The experimental indicators and segmentation results of the network proposed in this invention are compared with those of other networks. Figure 7 and Figure 8 (a)- Figure 8 As shown in (g), it can be seen from the figure that the evaluation indicators of the method proposed in the present invention are higher than those of the existing segmentation network, and the segmentation effect is closest to the original label of the landslide.

[0077] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned landslide image segmentation method based on high-low semantic information fusion are implemented.

[0078] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned landslide image segmentation method based on high-low semantic information fusion.

[0079] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process in the flowchart and the combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A device that specifies functions in a process or multiple processes.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

[0083] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A landslide image segmentation method based on high and low semantic information fusion, characterized in that: The steps include: Step 1, obtaining a landslide remote sensing image dataset and preprocessing the dataset, dividing the preprocessed dataset into a training set, a validation set, and a test set; Step 2, building a semantic segmentation network for landslide remote sensing images based on high and low semantic information fusion, including an encoder and a decoder, wherein the encoder includes a backbone network, an enhanced global information pooling module, and an enhanced target information attention module, and the decoder includes a first to second feature interaction module and a first to second multi-level semantic information fusion module; After the input image of the encoder passes through the backbone network, the backbone network outputs low-level landslide feature maps with image sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image respectively. The low-level landslide feature map with an image size of 1 / 32 of the input image passes through the enhanced global information pooling module and the enhanced target information attention module in turn to obtain the high-level landslide feature map D1; The low-level landslide feature maps with image sizes of 1 / 8 and 1 / 16 of the input image are stacked with each other through the first feature interaction module to generate feature map E1. D1 is superimposed with E1 after upsampling operation to output feature map F1. F1 and D1 are input to the first multi-level semantic information fusion module to output feature map G1; the low-level landslide feature maps with image sizes of 1 / 4 and 1 / 8 of the input image are stacked with each other through the second feature interaction module to generate feature map E2. G1 is superimposed with E2 after upsampling operation to output feature map F2. F2 and G1 are input to the second multi-level semantic information fusion module to output feature map G2; the low-level landslide feature map with image size of 1 / 4 of the input image is subjected to 1×1 convolution to generate feature map E3. E3 and G2 are fused and successively subjected to 1×1 convolution and LeakyReLU activation function to generate feature map F3. F3 is multiplied by E3 and successively subjected to 3×3 convolution and upsampling operation to obtain the output image of the decoder; The enhanced global information pooling module includes the second to fourth convolutional layers, the first to sixth dilated convolutional layers, the global average pooling layer and the strip pooling layer. The second to fourth convolutional layers are connected with batch normalization and ReLU activation functions, the first to sixth dilated convolutional layers are connected with batch normalization and ReLU activation functions, the convolution kernel sizes of the second to fourth convolutional layers are 1×1, the convolution kernel sizes of the first to sixth dilated convolutional layers are 3×3, the expansion rates of the first to third dilated convolutional layers are 1, 2 and 3 respectively, and the expansion rates of the fourth to sixth dilated convolutional layers are 1, 2 and 5 respectively; The backbone network outputs a low-level landslide feature map F with an image size of 1 / 32 of the input image. a As the input of the enhanced global information pooling module, the second convolutional layer outputs the feature map F b , F b The feature map F is generated by the third convolutional layer b1 , F b The feature map F is generated by the first, second and third hole convolution layers in sequence. b2 , F b The feature map F is generated by the fourth, fifth and sixth hole convolution layers in sequence. b3 , F b The feature map F is generated by the global average pooling layer b4 , F b The feature map F is generated by the strip pooling layer b5 , F b1 、F b2 、F b3 、F b4 and F b5 After cascading, and then with F a The fusion result is passed through the fourth convolutional layer to obtain the output F of the enhanced global information pooling module. d ; The structures of the first and second feature interaction modules are the same, both of which include the fifth and sixth convolutional layers, an enhanced target information attention module and a ReLU activation function. The fifth and sixth convolutional layers are both connected with batch normalization. The convolution kernel size of the fifth convolutional layer is 1×1, and the convolution kernel size of the sixth convolutional layer is 3×3. For the two inputs of each feature interaction module, the input F with the smallest image size 1-2 After upsampling, the sixth convolutional layer is used to obtain the feature map F 1-4 ; Enter another one into F 1-1 The feature map F is obtained through the fifth convolutional layer 1-3 , F 1-3 and F 1-4 The fusion result F 1-5 As the input of the enhanced target information attention module, the output of the enhanced target information attention module is activated by the ReLU function to obtain the output of each feature interaction module; The enhanced target information attention module includes the seventh to eighth convolutional layers, the first to second channel attention units and the spatial attention unit. The seventh to eighth convolutional layers are connected with batch normalization and LeakyReLU activation functions, and the convolution kernel sizes of the seventh to eighth convolutional layers are both 1×1. Enhance the input F of the attention module with target information 2-1 It is divided into three paths. In the first path, F 2-1 After passing through the first channel attention unit and the seventh convolutional layer, the feature map F is obtained. 2-2 ; In the second path, F 2-1 After the second channel attention unit and F 2-1 Multiply, and the result of the multiplication is added to F after passing through the spatial attention unit 2-2 Superposition, to obtain the feature map F with increased target information attention 2-3 ; In the third path, F 2-3 After the eighth convolutional layer, the output of the enhanced target information attention module is obtained; The structures of the first to second multi-level semantic information fusion modules are the same, and both include the first to second average pooling layers, the ninth to eighteenth convolutional layers, the first to second average pooling layers are connected with convolutional layers and batch normalization, the ninth to eighteenth convolutional layers are connected with batch normalization and ReLU activation functions, the convolution kernel sizes of the tenth, thirteenth, fourteenth, fifteenth, sixteenth and eighteenth convolutional layers are all 1×1, the convolution kernel sizes of the eleventh and seventeenth convolutional layers are all 3×3, the convolution kernel size of the ninth convolutional layer is 1×3, the convolution kernel size of the twelfth convolutional layer is 3×1, and the convolution kernel sizes of the convolutional layers connected after the first to second average pooling layers are all 1×1; For the two inputs of each multi-level semantic information fusion module, the input with the smallest image size is F 10 Upsampling is performed, and the upsampling results are successively passed through the first average pooling layer and the LeakyReLU activation function to obtain the feature map F 12 The up-sampling result is passed through the ninth convolutional layer to obtain the fourth output, and the fourth output is passed through the tenth, eleventh and twelfth convolutional layers respectively. The output results of the tenth, eleventh and twelfth convolutional layers are superimposed and then passed through the thirteenth convolutional layer to obtain the feature map F 11 ; F 11 and F 12 After superposition, we get the feature map F 13 ; Enter another one into F 20 After the fourteenth, fifteenth and sixteenth convolutional layers, the feature map F is obtained. 21 ; Another input F 20 The feature map F is obtained by the second average pooling layer and the LeakyReLU activation function. 22 ; F 22 and F 21 After superposition, we get the feature map F 23 ; F 23 and F 13 After fusion, the fusion results are sequentially passed through the 17th and 18th convolutional layers to obtain the output of each multi-level semantic information fusion module; Step 3, input the training set and the validation set into the landslide remote sensing image semantic segmentation network based on high and low semantic information fusion for training and validation, calculate the loss function and perform back propagation, update the network parameters, and obtain the trained optimal parameter network; Step 4: Input the test set into the trained optimal parameter network and output the segmentation effect diagram of the landslide image.

2. The landslide image segmentation method based on high and low semantic information fusion according to claim 1 is characterized in that: The specific process of step 1 is as follows: Obtain a landslide remote sensing image dataset, including landslide remote sensing images, label maps corresponding to landslide remote sensing images, and non-landslide remote sensing images; Perform data enhancement operations on the landslide remote sensing images in the dataset, including mirroring and flipping; The data set after data augmentation operation is divided into training set, validation set and test set according to preset proportions.

3. The landslide image segmentation method based on high and low semantic information fusion according to claim 1 is characterized in that: The backbone network includes an expanded convolution layer, a depth-separable convolution layer, an SE attention module and a first convolution layer. The input of the backbone network passes through the expanded convolution layer and the depth-separable convolution layer in sequence to obtain a first output. The first output passes through the SE attention module to obtain a second output. The first output is multiplied by the second output and passed through the first convolution layer to obtain a third output. The third output is multiplied by the input of the backbone network to obtain the output of the backbone network. The convolution kernel size of the expanded convolution layer is 1×1, the convolution kernel size of the depth-separable convolution layer is 3×3, and the convolution kernel size of the first convolution layer is 1×1. The depthwise separable convolution layer includes a depthwise convolution layer and a pointwise convolution layer connected in sequence, and the depthwise convolution layer and the pointwise convolution layer are both connected with batch normalization and ReLU activation functions; the convolution kernel size of the pointwise convolution layer is 1×1×M, where M is the input dimension; The SE attention module includes a global average pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence, and the first fully connected layer is connected with a ReLU activation function, and the second fully connected layer is connected with a hwish activation function.

4. The landslide image segmentation method based on high and low semantic information fusion according to claim 1 is characterized in that: In step 3, the loss function is obtained by adding the Diess loss function and the cross entropy loss function, and the formula is as follows: Among them, L D and L C Denote the Dice loss function and the cross entropy loss function respectively, N represents the total number of target pixels, y i and They represent the label value and predicted value of the training network for the target pixel i respectively.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the landslide image segmentation method based on high-low semantic information fusion as described in any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the landslide image segmentation method based on high-low semantic information fusion as described in any one of claims 1 to 4 are implemented.

Images