Landslide interpretation semantic segmentation method, system and device based on hybrid attention and medium

Abstract

The application discloses a landslide interpretation semantic segmentation method, system and device based on mixed attention and a medium, and comprises the following steps: a mixed attention module adopts a parallel double-branch structure, and a space-channel bidirectional interaction module is used to combine different features extracted from two paths; so that the model can simultaneously learn the spatial features and channel features of the input image, and can accurately identify the position where the mountain landslide occurs in a complex geographical environment. Meanwhile, the application uses a multi-scale feature mutual compensation module to fuse the output of each stage encoder into the coarse-grained features of the next stage through a depth separable convolution as fine-grained features, and uses an upsampling module before the input decoder to assign the coarse-grained features to the fine-grained features of the next stage, so that the features of each stage finally have a global semantic view, have better generalization for different terrains, and achieve the best effect in landslide segmentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and deep learning, and relates to a landslide interpretation semantic segmentation method, system and device based on hybrid attention and a medium. BACKGROUND

[0002] Early landslide detection methods mainly rely on traditional machine learning, and deep neural networks have not been applied in this field. In addition to landslide area detection, landslide susceptibility detection is also crucial. The goal of disaster and risk probability mapping is to identify and describe disaster-prone areas in space, and to conduct targeted research to analyze potential risks. In order to analyze risks, it is necessary to study the causes, characteristics and impacts of past disasters, and then link all factors to the current situation to generate a disaster-prone probability map. Almost all machine learning methods require labeled data sets for training, validation and testing, however, time-consuming and labor-intensive problems are inevitable in traditional landslide mapping methods.

[0003] In recent years, with the rapid development of deep learning technology, it has shown superior performance over traditional machine learning methods in various tasks, so deep learning models are becoming more and more popular. However, the widespread application of deep learning models in landslide interpretation also brings new challenges. One major challenge is the difficulty of generalization when applying the model to data with different geographical conditions. Another basic challenge is the lack of public data sets for landslides, which hinders the widespread development of landslide detection and affects the applicability and accuracy of deep learning in the field of landslides. SUMMARY

[0004] The purpose of the present application is to solve the problem of difficulty in generalization of deep learning in different geographical conditions, which affects the overall applicability and accuracy in the prior art, and to provide a landslide interpretation semantic segmentation method, system, device and medium based on hybrid attention.

[0005] To achieve the above purpose, the following technical solutions are adopted:

[0006] The landslide interpretation semantic segmentation method based on hybrid attention includes:

[0007] Step 1: Obtain a landslide data set and divide it into a training set and a test set;

[0008] Step 2: Construct a feature extraction module based on hybrid attention, extract spatial features based on cross-window attention, and extract channel features based on continuous stacked depth separable convolution;

[0009] Step 3: a space-channel bidirectional interaction module is constructed to interact the obtained space feature information and channel feature information, so as to obtain fused features;

[0010] Step 4: the fused features are encoded based on the multi-scale fusion module to obtain information of different scales and different granularities at the current stage; and the information of different scales and different granularities at the current stage is fused with features of other stages;

[0011] Step 5: the fused features are decoded based on the feature pyramid decoder to output feature images;

[0012] Step 6: steps 2 to 5 are repeatedly trained based on the training set to obtain an optimized semantic segmentation model;

[0013] Step 7: the region of the mountain landslide of the test set is identified based on the optimized semantic segmentation model to obtain a final semantic segmentation result.

[0014] Further improvements of the application are as follows:

[0015] Further, the division ratio of the training set and the test set is randomly divided or divided at a preset ratio; the preset ratio is artificially set.

[0016] Further, a hybrid attention module is constructed, space features are extracted based on cross window attention, and channel features are extracted based on continuous stacked depth separable convolution, specifically:

[0017] The cross window attention is designed to extract space features to obtain a branch feature representation; the depth separable convolution is used to extract channel features; each hybrid attention module includes layer normalization, hybrid attention, feedforward neural network and residual link; wherein, the hybrid attention module is:

[0018]

[0019]

[0020] wherein, X l+1 respectively represent the output of the entire hybrid attention module and the output of the feedforward neural network; Hybrid represents a function of outputting fused channel features and space features, and LN represents a layer normalization function;

[0021] The cross window attention in the hybrid attention is:

[0022]

[0023] V * = V x sigma (C2(()))

[0024] wherein, H-Attention represents the attention in the horizontal direction, Q, K and V represent the query, key value and value matrix respectively, sigma represents the sigmoid activation function, C2T represents the channel-space interaction attention module in the next section, CFE represents the channel feature extraction module, and V* represents the value matrix after channel-space interaction;

[0025] The cross window attention is summarized as follows:

[0026]

[0027]

[0028] wherein, head represents the multi-head attention after linear projection, K is the dimension number of projection, and H / V-Attention represents the attention in the horizontal direction and the vertical direction respectively.

[0029] Further, a space-channel bidirectional interaction module is constructed to interact the obtained spatial feature information and channel feature information to obtain fused features, and the specific process is as follows:

[0030] The space-channel bidirectional interaction module includes a channel-space interaction module and a space-channel interaction module; the channel-space interaction module is formed by two groups of 1x1 deep convolution kernels, batch normalization operations and ReLU nonlinear activation functions stacked in sequence; specifically, the channel feature extraction module obtains an output with a channel number of C, and the channel dimension is still maintained after the output passes through the channel-space interaction module; based on the nonlinear activation function sigmoid, the channel feature is assigned as a channel factor to each channel dimension of the value matrix of the cross window attention, and the learning result of the attention module has both spatial features and channel features; the channel feature extraction module is a deep separable convolution stacked in sequence; relatively, the space-channel interaction module transmits the spatial features learned by the final attention mechanism to the result of the channel feature extraction module as a feature factor; finally, the features extracted by the double-branch structure are added to obtain the fused features.

[0031] Further, a multi-scale feature fusion module is constructed, and the specific process is as follows: the relationship between the spatial features and the channel features inside the feature image is completely decoupled, and a skip connection structure is used for connection, wherein the skip connection structure includes a first deep separable convolution module and a second deep separable convolution module; the first deep separable convolution module is used for feature extraction between the encoders-decoders in the same stage; and the second deep separable convolution module is used for feature extraction between the encoders-decoders in different stages.

[0032] The fine-grained features extracted by the previous stage encoder are fused with the coarse-grained features extracted by the next stage encoder, so that the bottleneck part of the network in each stage contains multi-scale and multi-granularity semantic information of adjacent stages when the first deep separable convolution module is used for feature extraction, and when the first deep separable convolution module is used for feature extraction, the input of the first deep separable convolution module contains all the different granularity semantic information when it is passed to the last stage.

[0033] The multi-scale feature fusion module further comprises an upsampling fusion module; the input of the upsampling fusion module is the feature extracted by the first deep separable convolution module, and the output of the upsampling fusion module aligns the spatial size of the feature map with the next stage through bilinear interpolation, and finally fuses the current feature with the coarse-grained original semantic feature.

[0034] Further, the first deep separable convolution module and the second deep separable convolution module are specifically:

[0035] The step length in the deep convolution of the first deep separable convolution module is 1; the input feature image passes through the first deep separable convolution module, and the channel number and spatial dimension of the input and output do not change; the step length in the deep convolution of the second deep separable convolution module is set to 2, and the channel number of the input feature image is halved and the spatial dimension is doubled after passing through the second deep separable convolution module.

[0036] Further, the fused features are decoded based on a feature pyramid decoder to output a feature image, specifically: the feature pyramid decoder comprises a 3x3 convolution kernel, a grouped normalization, a ReLU activation function, and a 2x bilinear interpolation upsampling; the input of the feature pyramid decoder includes four stages; the input of the fourth stage is a feature map after 32 times of downsampling, and the required feature map size is obtained after three times of upsampling; the downsampling rates of the third stage, the second stage and the first stage are 16, 8 and 4 respectively, so the number of upsampling modules decreases to 0 in turn, and finally each stage of the decoder produces a feature map with an input of 1 / 4, the four outputs are added element by element, and a final feature image is obtained through a 4x upsampling fusion module.

[0037] The landslide interpretation semantic segmentation system based on hybrid attention comprises:

[0038] The division module acquires a landslide data set and divides it into a training set and a test set;

[0039] The extraction module extracts spatial features based on cross-window attention and extracts channel features based on a continuous stack of deep separable convolution;

[0040] An interaction module is configured to interact the acquired spatial feature information and channel feature information to obtain fused features;

[0041] An encoding module is configured to encode the fused features based on the multi-scale fusion module to obtain information of different scales and different granularities at a current stage, and perform feature fusion between the information of different scales and different granularities at the current stage and features of other stages;

[0042] A decoding module is configured to decode the fused features based on a feature pyramid decoder to output a feature image;

[0043] A training module is configured to repeatedly train based on a training set to obtain an optimized semantic segmentation model;

[0044] An identification module is configured to perform regional identification of mountain landslide on a test set based on the optimized semantic segmentation model to obtain a final semantic segmentation result.

[0045] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0046] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] In the present application, the mixed attention module adopts a parallel double-branch structure, and a space-channel bidirectional interaction module is used to combine the different features extracted from two paths; the model can learn the spatial features and channel features of the input image at the same time, thereby better performing regional segmentation on the landslide, and the position of the mountain landslide can still be accurately identified in a complex geographical environment.

[0049] Further, the present application uses a multi-scale feature mutual compensation module to fuse the output of each stage encoder into the coarse-grained features of the next stage through a depth separable convolution as fine-grained features, and uses an upsampling module before the input decoder to assign the coarse-grained features to the fine-grained features of the next stage, so that the features of each stage have a global semantic view, not only the input of the decoder of each stage of the model has small size and coarse-grained semantic information, but also large size and fine-grained semantic information, the model has a larger semantic view and pays attention to long-range dependence, has better generalization for different terrains, and the effect on landslide segmentation is the best.

[0050] Further, the final feature map after the feature pyramid decoder of the present application contains not only coarse-grained and fine-grained semantic information but also multi-scale global information, so that the method has better effect on the segmentation task of landslides. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0052] Figure 1 is a flowchart of another landslide interpretation semantic segmentation method based on hybrid attention of the present application;

[0053] Figure 2 is a structural diagram of a landslide interpretation semantic segmentation system based on hybrid attention of the present application;

[0054] Figure 3 is a flowchart of another landslide interpretation semantic segmentation method based on hybrid attention of the present application;

[0055] Figure 4 is a landslide interpretation semantic segmentation architecture diagram based on hybrid attention of the present application;

[0056] Figure 5 is a structural diagram of a multi-scale feature fusion module of the present application;

[0057] Figure 6 is a structural diagram of a depth separable convolution of the present application;

[0058] Figure 7 is another structural diagram of a multi-scale feature fusion module of the present application;

[0059] Figure 8 is a structural diagram of a depth separable convolution module used for feature extraction and fusion at the bottleneck part of the network of the present application;

[0060] Figure 9 is a structural diagram of a semantic segmentation feature pyramid decoder module of the present application;

[0061] Figure 10 is a comparison diagram of landslide areas identified by a commonly used reference method and the method proposed by the present application. DETAILED DESCRIPTION

[0062] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0064] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0065] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is used, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0066] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0067] In the description of the embodiments of the present application, it should also be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense, for example, they can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0068] The present application will be described in further detail below with reference to the drawings:

[0069] Referring toFigure 1 The application discloses a landslide interpretation semantic segmentation method based on hybrid attention, comprising the following steps:

[0070] S101: Obtain a mountain landslide data set and divide it into a training set and a test set.

[0071] The division ratio of the training set and the test set is randomly divided or divided at a preset ratio; the preset ratio is artificially set

[0072] S102: Construct a feature extraction module based on hybrid attention, extract spatial features based on cross window attention, and extract channel features based on continuous stacked depth separable convolution.

[0073] Design cross window attention to extract spatial features to obtain a branch of feature representation; depth separable convolution extracts channel features; each hybrid attention module includes layer normalization, hybrid attention, feedforward neural network and residual link; wherein the hybrid attention module is:

[0074]

[0075]

[0076] wherein, X l+1 respectively represent the output of the entire hybrid attention module and the output of the feedforward neural network; Hybrid represents a function of fusing channel features and spatial features, and LN represents a layer normalization function;

[0077] The cross window attention in the hybrid attention is:

[0078]

[0079] V * = V x sigma (C2T (H-Attention (Q, K, V)))

[0080] wherein, H-Attention represents horizontal attention, Q, K and V represent query, key value and value matrix respectively, sigma represents a sigmoid activation function, C2T represents a channel-space interaction attention module in the next section, CFE represents a channel feature extraction module, and V* represents a value matrix after channel-space interaction;

[0081] The cross window attention is summarized as follows:

[0082]

[0083]

[0084] Wherein, head represents multi-head attention after linear projection, K is the dimension number of projection, and H / V-Attention represents attention in horizontal direction and vertical direction respectively.

[0085] S103: A space-channel bidirectional interaction module is constructed to interact the obtained space feature information and channel feature information to obtain fused features.

[0086] The space-channel bidirectional interaction module comprises a channel-space interaction module and a space-channel interaction module; the channel-space interaction module is formed by two groups of 1*1 deep convolution kernels, batch normalization operations and ReLU nonlinear activation functions stacked in sequence; specifically, the channel feature extraction module obtains an output with a channel number of C, the channel dimension is still maintained after the output passes through the channel-space interaction module, the channel feature is assigned to each channel dimension of the value matrix of the cross window attention based on the nonlinear activation function sigmoid as a channel factor, and the learning result of the attention module has both space features and channel features; the channel feature extraction module is a deep separable convolution stacked in sequence; relatively, the space-channel interaction module transmits the space features learned by the final attention mechanism to the result of the channel feature extraction module as a feature factor; finally, the features extracted by the double-branch structure are added to obtain fused features.

[0087] S104: The fused features are encoded based on a multi-scale fusion module to obtain information of different scales and different granularities at the current stage; and the information of different scales and different granularities at the current stage is fused with features of other stages.

[0088] The multi-scale feature fusion module is specifically: the relationship between the space features and the channel features inside the feature image is completely decoupled, and a cross-connection structure is used for connection, the cross-connection structure comprises a first deep separable convolution module and a second deep separable convolution module; the first deep separable convolution module is used for extracting features between encoders-decoders of the same stage; the second deep separable convolution module is used for extracting features between encoders-decoders of different stages.

[0089] The fine-grained features extracted by the previous stage encoder are fused with the coarse-grained features extracted by the next stage encoder, so that the input of the bottleneck part of the network in each stage when the first deep separable convolution module is used for feature extraction contains multi-scale and multi-granularity semantic information of adjacent stages; when transmitted to the last stage, the input of the first deep separable convolution module contains semantic information of all different granularities.

[0090] The multi-scale feature fusion module further comprises: an up-sampling fusion module; the input of the up-sampling fusion module is the feature extracted by the first deep separable convolution module, the output of the up-sampling fusion module aligns the spatial size of the feature map with the next stage through bilinear interpolation, and finally the current feature is fused with the coarse-grained original semantic feature. The step length in the depth-wise convolution of the first deep separable convolution module is 1; the input feature image passes through the first deep separable convolution module, and the channel number and spatial dimension of the input and output do not change; the step length in the depth-wise convolution of the second deep separable convolution module is set to 2, and after the input feature image passes through the second deep separable convolution module, the channel number is halved and the spatial dimension is doubled.

[0091] S105: decoding the fused feature based on the feature pyramid decoder, and outputting a feature image.

[0092] The feature pyramid decoder comprises a 3*3 convolution kernel, a group normalization, a ReLU activation function and a 2 times bilinear interpolation up-sampling; the input of the feature pyramid decoder comprises four stages; the input of the fourth stage is a feature map after 32 times down-sampling, and the required feature map size is obtained through three times up-sampling; the down-sampling multiples of the third stage, the second stage and the first stage are 16, 8 and 4 respectively, so that the number of up-sampling modules is reduced to 0 in turn, and finally a feature map with 1 / 4 input is generated by the decoder of each stage, the four outputs are element-wise added, and a final feature image is obtained through a 4 times up-sampling fusion module.

[0093] S106: repeatedly training S102 to S105 based on the training set, and obtaining an optimized semantic segmentation model.

[0094] S107: performing regional identification of mountain landslide based on the optimized semantic segmentation model, and obtaining a final semantic segmentation result.

[0095] Referring to Figure 2 , the application discloses a landslide interpretation semantic segmentation system based on mixed attention, comprising:

[0096] A division module, which acquires a mountain landslide data set and divides it into a training set and a test set;

[0097] An extraction module, which extracts spatial features based on cross-window attention and extracts channel features based on continuous stacked deep separable convolution;

[0098] An interaction module, which interacts the acquired spatial feature information and channel feature information to obtain fused features;

[0099] An encoding module encodes the fused features based on the multi-scale fusion module to obtain information of different scales and different granularities at the current stage, and fuses the information of different scales and different granularities at the current stage with features of other stages;

[0100] A decoding module decodes the fused features based on a feature pyramid decoder to output a feature image.

[0101] A training module repeatedly trains based on a training set to obtain an optimized semantic segmentation model.

[0102] An identification module identifies a mountain landslide area based on the optimized semantic segmentation model to obtain a final semantic segmentation result.

[0103] Referring to Figure 3 and Figure 4 , the embodiment of the present application provides a mountain landslide semantic segmentation method based on a hybrid attention mechanism, which comprises the following steps:

[0104] Step S1: Obtain a mountain landslide data set, and divide it into a training set and a test set in proportion; the training set is used for training of a model, and the test set is used for testing of the trained model. The above data set division can be random division or division according to a preset proportion, and the embodiment is not limited.

[0105] Step S2: extract spatial features by using cross window attention to obtain feature representation of a branch; extract channel features by using depth separable convolution; each hybrid attention module is composed of layer normalization, hybrid attention, feedforward neural network and residual link. The process of the hybrid attention module is represented as:

[0106]

[0107]

[0108] wherein and X l+1 respectively represent the output of the whole hybrid attention module and the output of the feedforward neural network. Hybrid represents a function of fusing channel features and spatial features, and LN represents a layer normalization function.

[0109] The cross window attention in the hybrid attention is represented as:

[0110]

[0111] V * = V x sigma (C2(()))

[0112] Wherein, H-Attention represents the attention in the horizontal direction, Q, K and V represent the query, key value and value matrix respectively, sigma represents the sigmoid activation function, C2T represents the channel-space interaction attention module in the next section, CFE represents the channel feature extraction module of the left branch, and V* represents the value matrix after channel-space interaction.

[0113] Referring to Figure 5 The summary of cross window attention is represented as:

[0114]

[0115]

[0116] Wherein, head represents the multi-head attention after linear projection, K is the dimension number of projection, and H / V-Attention represents the attention in the horizontal and vertical directions respectively.

[0117] Step S3: The channel-space interaction module is stacked by two groups of 1x1 deep convolution kernels, batch normalization operation (BN) and ReLU nonlinear activation function. Referring to Figure 6 and Figure 7 Specifically, the channel feature extraction module obtains an output with a channel number of C, the channel dimension is still maintained after the channel-space interaction module, and then a nonlinear activation function sigmoid is used to assign the channel feature as a channel factor to each channel dimension of the value matrix of the cross window attention, that is, the two-dimensional matrix multiplied by each dimension is different. The learning result of the final attention module is that it has both spatial and channel features. The ablation experiment proves that this method is simple and effective.

[0118] In contrast, the purpose of the space-channel interaction module is to pass the spatial dimension features learned by the final attention mechanism to the result of the channel feature extraction module as a feature factor. Finally, the features extracted by the double-branch structure are added to obtain the fused features.

[0119] Step S4: The features obtained by the encoder part are used in the bottleneck part of the model using a multi-scale feature fusion module for further feature expression optimization. The multi-scale feature fusion module includes the following steps:

[0120]

[0121]

[0122]

[0123] Wherein, The output of the bottleneck part of the representative model at the i-th stage, which is also the input of the decoder of the i-th stage. Figure 8 The structure of the first deep separable convolution module DWSC and the second deep separable convolution module DWSC* respectively represents the deep separable convolution module between the same stages and the deep separable convolution module between adjacent stages, and each deep separable convolution module is composed of a channel-wise convolution, a batch normalization, a ReLU, a point-wise convolution, a batch normalization and a ReLU activation function, and the main difference between them is that the convolution step is 1 and 2. Obviously The input of the semantic segmentation pyramid module is composed of two parts, one part is from the encoder output result of the current stage, and the other part is from the encoder output of the last stage after a deep separable convolution with a step of 2.

[0124] Figure 9 The semantic segmentation pyramid module used in the decoder part of the model of the application, the leftmost picture in the figure respectively represents the last decoder input of the four different stages, and the score in the figure represents the down-sampling ratio of the encoder.

[0125] Step S5: Each stage adopts a semantic segmentation pyramid decoder module. Each decoder contains a module composed of a 3x3 convolution kernel, a group normalization, a ReLU activation function and a 2 times bilinear interpolation up-sampling. The final input of the fourth stage is a feature map after 32 times down-sampling, which needs to be up-sampled three times to obtain the required feature map size. The down-sampling ratios of the third, second and first stages are 16, 8 and 4 respectively, so the number of up-sampling modules decreases to 0 in turn, and finally each stage of the decoder will produce a feature map with 1 / 4 input. After an element-wise addition operation is performed on the four outputs, a 4 times up-sampling module is used to obtain the final feature map output.

[0126] For this structure, both the parameter amount and the operation amount are much lighter than the traditional U-shaped structure decoder and the ordinary FPN structure. Since the input of the decoder is the fusion of the output of the encoder of the same stage after DWSC feature extraction and the original semantic information of the previous stage with a coarse granularity, the final feature map after the feature pyramid decoder contains not only the coarse-grained and fine-grained semantic information but also the multi-scale global information, so that the method has better effect on the segmentation task of landslides.

[0127] Step S6: According to the batch-based stochastic gradient descent method, the landslide semantic segmentation model is trained using the training set.

[0128] Step S7: Using the trained model, the landslide semantic segmentation is performed on the test set of landslides to obtain the landslide area recognition result.

[0129] The present application is fully and comprehensively tested on a landslide data set, and the experimental results show that the present application significantly improves the model's landslide area recognition ability.

[0130] As shown in Table 1: Precision, Recall, F1-score, mACC, mIoU numerical comparison table on the landslide face data set. The present application compares the Precision, Recall, F1-score, mACC, mIoU values on the landslide data set with other benchmark methods. The benchmark methods compared include DPT, Twins-svt, Segformer, SWin-Transformer and CSWin-Transformer. As can be seen from the quantitative results of Table 1, the three indicators of the dual-path face image super-resolution reconstruction model DPMF Net of the present application are significantly higher than those of other methods.

[0131] Table 1: Precision, Recall, F1-score, mACC, mIoU numerical comparison table on the landslide face data set

[0132]

[0133]

[0134] Reference Figure 10 , Figure 10 The landslide areas recognized by the above-mentioned benchmark methods and the landslide semantic segmentation model HA-Net based on the hybrid attention mechanism proposed by the present application are shown. As can be seen from Figure 9 It can be seen that the landslide semantic segmentation model HA-Net based on the hybrid attention mechanism proposed by the present application has very good segmentation effect on landslides of different scales in various geographical environments, with clear edges and accurate recognition.

[0135] The terminal device provided by an embodiment of the present application. The terminal device of the embodiment includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in each of the above method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the above device embodiments are implemented.

[0136] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application.

[0137] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The terminal device can include, but is not limited to, a processor and a memory.

[0138] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0139] The memory can be used to store the computer program and / or modules, and the processor can realize various functions of the terminal device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory.

[0140] The modules / units integrated in the terminal device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution medium, etc. It should be noted that the contents included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0141] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A semantic segmentation method for landslide interpretation based on hybrid attention, characterized in that, include: Step 1: Obtain the landslide dataset and divide it into training and test sets; Step 2: Construct a feature extraction module based on hybrid attention, extract spatial features based on cross-window attention, and extract channel features based on continuously stacked depthwise separable convolutions; Step 3: Construct a spatial-channel bidirectional interaction module to interact the acquired spatial feature information and channel feature information to obtain the fused features, specifically: The spatial-channel bidirectional interaction module includes a channel-space interaction module and a spatial-channel interaction module. The channel-space interaction module consists of two sets of 1×1 depthwise convolutional kernels, batch normalization operations, and ReLU nonlinear activation functions stacked consecutively. Specifically, the channel feature extraction module obtains an output with C channels. After passing through the channel-space interaction module, the channel dimension remains unchanged. Based on the nonlinear activation function sigmoid, the channel features are assigned as channel factors to each channel dimension of the cross-window attention value matrix. The learning result of the attention module has both spatial and channel features. The channel feature extraction module is a continuously stacked depthwise separable convolution. In contrast, the spatial-channel interaction module passes the spatial features learned by the final attention mechanism as feature factors to the result of the channel feature extraction module. Finally, the features extracted by the dual-branch structure are summed to obtain the fused features. Step 4: Encode the fused features based on the multi-scale fusion module to obtain information at different scales and granularities in the current stage; and fuse the information at different scales and granularities in the current stage with features from other stages; Step 5: Decode the fused features based on the feature pyramid decoder and output the feature image; Step 6: Repeat steps 2 to 5 based on the training set to obtain the optimized semantic segmentation model; Step 7: Based on the optimized semantic segmentation model, perform landslide area identification on the test set to obtain the final semantic segmentation result.

2. The landslide interpretation semantic segmentation method based on hybrid attention according to claim 1, characterized in that, The division ratio between the training set and the test set is either random or based on a preset ratio; the preset ratio is set manually.

3. The landslide interpretation semantic segmentation method based on hybrid attention according to claim 2, characterized in that, The construction of the feature extraction module based on hybrid attention, which extracts spatial features based on cross-window attention and extracts channel features based on continuously stacked depthwise separable convolutions, specifically involves: The design incorporates cross-window attention to extract spatial features, resulting in a feature representation for one branch; depthwise separable convolution extracts channel features; each hybrid attention module includes layer normalization, hybrid attention, a feedforward neural network, and residual connections; the hybrid attention module is as follows: in, , These represent the output of the entire hybrid attention module and the output of the feedforward neural network, respectively; Hybrid represents the function that fuses channel features and spatial features, and LN represents the layer normalization function. Cross-window attention in hybrid attention is: Here, H-Attention represents horizontal attention, and Q, K, and V represent the query, key, and value matrix, respectively. C represents the sigmoid activation function, C2T represents the channel-space interaction attention module in the next section, CFE represents the channel feature extraction module, and V* represents the value matrix after channel-space interaction. The attention required for cross windows is summarized as follows: Here, head represents multi-head attention after linear projection, K is the number of projection dimensions, and H / V-Attention represents attention in the horizontal and vertical directions, respectively.

4. The landslide interpretation semantic segmentation method based on hybrid attention according to claim 3, characterized in that, The multi-scale fusion module specifically comprises: the spatial features and channel features are completely decoupled within the feature image and connected using a cross-connection structure, which includes a first depthwise separable convolutional module and a second depthwise separable convolutional module; the first depthwise separable convolutional module is used for feature extraction between encoders and decoders at the same stage; the second depthwise separable convolutional module is used for feature extraction between encoders and decoders at different stages. The fine-grained features extracted by the encoder in the previous stage are fused with the coarse-grained features extracted by the encoder in the next stage. This ensures that the input of the bottleneck part of the network in each stage when using the first deep separable convolutional module for feature extraction contains multi-scale and multi-granular semantic information from adjacent stages. When the signal is passed to the last stage, the input of the first deep separable convolutional module contains semantic information of all different granularities. The multi-scale fusion module further includes: an upsampling fusion module; The input to the upsampling fusion module is the feature extracted by the first depthwise separable convolution module. The output of the upsampling fusion module aligns the spatial size of the feature map with the next stage through bilinear interpolation. Finally, the current feature is fused with the coarse-grained original semantic features.

5. The landslide interpretation semantic segmentation method based on hybrid attention according to claim 4, characterized in that, The first depthwise separable convolutional module and the second depthwise separable convolutional module are specifically as follows: The stride in the depthwise convolution of the first depthwise separable convolution module is 1; the number of input and output channels and spatial dimension of the input feature image do not change after passing through the first depthwise separable convolution module; the stride in the depthwise convolution of the second depthwise separable convolution module is set to 2, and the number of input channels is halved while the spatial dimension is doubled after passing through the second depthwise separable convolution module.

6. The landslide interpretation semantic segmentation method based on hybrid attention according to claim 5, characterized in that, The feature pyramid decoder decodes the fused features and outputs a feature image. Specifically, the feature pyramid decoder includes a 3×3 convolution kernel, group normalization, ReLU activation function, and 2x bilinear interpolation upsampling. The input of the feature pyramid decoder includes four stages; the fourth stage input is a feature map downsampled by 32 times, which is upsampled three times to obtain the required feature map size; the downsampling ratios of the third, second and first stages are 16, 8 and 4 respectively, so the number of upsampling modules is reduced to 0 in sequence. Finally, the decoder of each stage produces a feature map with 1 / 4 times the input. The four outputs are added element-wise and then fused by a 4x upsampling module to obtain the final feature image.

7. A landslide interpretation semantic segmentation system based on hybrid attention, based on the landslide interpretation semantic segmentation method based on hybrid attention as described in claim 1, characterized in that, include: A partitioning module acquires the landslide dataset and divides it into a training set and a test set; The extraction module extracts spatial features based on cross-window attention and extracts channel features based on continuously stacked depthwise separable convolutions. An interaction module that interacts with the acquired spatial feature information and channel feature information to obtain fused features; The encoding module encodes the fused features based on the multi-scale fusion module to obtain information at different scales and granularities at the current stage. And feature fusion is performed between information of different scales and granularities at the current stage and features from other stages; The decoding module decodes the fused features based on the feature pyramid decoder and outputs a feature image; The training module performs repeated training based on the training set to obtain the optimal semantic segmentation model; The identification module identifies landslide areas on the test set based on an optimized semantic segmentation model, and obtains the final semantic segmentation result.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images