A landslide extraction method based on multi-task learning and multi-source data feature fusion

Abstract

This invention relates to a landslide extraction method based on multi-task learning and multi-source data feature fusion, belonging to the field of remote sensing image processing. First, multi-source data containing landslide features is acquired and preprocessed. Then, the preprocessed data is input into a defined BFM-UNet model, which includes a branch-fusion encoder and a multi-task decoder. The branch-fusion encoder extracts features from different source data through two branches and fuses these features using cross-stitch units, thereby effectively utilizing information from the multi-source data. The multi-task decoder adds the task of identifying widely existing and easily identifiable non-landslide features within the target area to assist landslide extraction, and leverages the advantages of multi-task learning by learning optimal sharing between tasks through cross-stitch units. Finally, the trained model is used to extract landslides, obtaining a landslide distribution map. Experimental results show that the landslide extraction method proposed in this invention can effectively improve the accuracy of landslide extraction and reduce the false detection rate.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and relates to a landslide extraction method based on multi-task learning and multi-source data feature fusion. Background Technology

[0002] Landslides are frequent natural disasters with powerful explosive force and destructive power, causing serious damage to people's lives and property. Rapid extraction of landslide information is crucial for disaster emergency response and land use planning. Traditional landslide extraction methods, primarily based on on-site investigation and measurement, while offering high mapping accuracy and detailed geological information, are time-consuming, labor-intensive, and inherently dangerous.

[0003] With the development of remote sensing technology, remote sensing data has been widely used in landslide extraction. Visual interpretation is a basic landslide extraction method based on remote sensing images. This method relies on the professional knowledge and experience of interpreters to identify landslide areas, offering high accuracy but also being highly subjective and difficult to handle large-scale data. Furthermore, remote sensing-based automatic landslide extraction methods can be broadly categorized into feature-threshold-based, shallow machine learning-based, and deep learning-based methods according to their technical principles. Feature-threshold-based methods construct feature rule sets by determining appropriate thresholds to achieve landslide extraction; however, finding suitable thresholds is difficult and their universality is limited. Machine learning-based methods use machine learning algorithms for classification to achieve landslide extraction. These methods have low data and computational resource requirements and high computational efficiency, but they are heavily dependent on feature selection, and their performance is limited by the expressive power of the machine learning model. In contrast, deep learning-based methods have shown great potential and advantages. UNet, due to its simple, flexible, and easily scalable architecture and wide applicability, has been widely used in landslide extraction based on remote sensing images. In specific application scenarios for landslide extraction using remote sensing images, UNet is often modified and optimized accordingly. Deep learning methods can automatically extract features from raw data, enabling end-to-end landslide extraction and making them more efficient for large-scale landslide detection.

[0004] Deep learning methods have demonstrated significant potential and advantages in landslide extraction. However, the unique characteristics of landslides make landslide identification generally more challenging, and existing methods still face several challenges. Due to their complex structure and large number of parameters, the good performance of deep learning models depends on a large amount of high-quality data samples. However, high-quality landslide data samples are scarce, leading to the problem of small-sample learning and limiting performance. Furthermore, previous studies often focused only on the characteristics of the landslide itself, simply treating non-landslide areas as background, thus ignoring information that the background areas might contain that could be helpful for landslide identification. In addition, because the slope area is similar in texture and color to the surrounding bare soil and rocks, has irregular shapes, and image quality is affected by factors such as lighting, shadows, clouds, fog, and vegetation, using only optical remote sensing imagery for landslide extraction is not ideal. Although some existing studies have considered combining multi-source data for landslide extraction, simply splicing or adding data often fails to fully utilize the unique information of each data type or to deeply explore the potential correlations between data.

[0005] To address the above issues, the method is optimized from two aspects: combining multi-task learning and fully utilizing multi-source data. Firstly, an identification task is added to identify non-landslide features that are widely present in the target area and are easier to identify than landslides, thus assisting landslide extraction. This allows the model to more effectively utilize feature information from the background (non-landslide areas) during landslide extraction. Simultaneously, multi-task learning enhances the network's feature mining capabilities and reduces its dependence on data. Secondly, the network structure is optimized to effectively extract and fuse feature information from multi-source data, ensuring that the model fully utilizes the information contained in the multi-source data during landslide extraction, further enhancing model performance. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a landslide extraction method based on multi-task learning and multi-source data feature fusion.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A landslide extraction method based on multi-task learning and multi-source data feature fusion includes the following steps:

[0009] S1: Data acquisition, including relevant multi-source data and vector label data, where the vector label data includes landslide labels and labels related to auxiliary tasks;

[0010] S2: Preprocess the acquired data. For relevant multi-source data, noise and interference need to be eliminated as much as possible. For relevant label data, convert it from vector to raster. Finally, ensure that all data have the same resolution and are spatially aligned.

[0011] S3: Construct a multi-task deep learning model BFM-UNet (Branch-Fusion Multi-task UNet) for landslide extraction using multi-source data. This model includes a branch-fusion encoder and a multi-task decoder.

[0012] S4: Training the model, including calculating the loss function and setting the relevant hyperparameters. During training, a joint loss function is used for each task, which combines weighted cross-entropy loss and Dice loss. In multi-task learning, learnable and dynamically changing task loss weights are used. In addition, an appropriate batch size and training epochs are set, and SGD (Stochastic Gradient Descent) is selected as the optimization algorithm with a dynamically changing learning rate.

[0013] S5: Based on the optical remote sensing images and topographic data of the target area after the preprocessing in S2, landslides are extracted using the trained BFM-UNet model to obtain a landslide distribution map.

[0014] Furthermore, in S1, the multi-source data to be acquired mainly includes optical remote sensing images and digital elevation models. Other remote sensing data containing landslide characteristic information, such as SAR images, may also be added as appropriate. In addition, landslide vector labels and auxiliary task vector labels need to be acquired. The auxiliary task selects ground features that are widely present in the area and are easier to identify than landslides as targets.

[0015] Furthermore, in S2, the preprocessing of multi-source data mainly includes: performing atmospheric correction on optical remote sensing images; selecting to export all or only some channels of data as needed, such as RGB three channels; performing terrain analysis based on DEM data to obtain slope and aspect data; if there is other data, such as SAR images, radiometric correction, denoising, and terrain correction are required; to ensure the resolution uniformity of multi-source data, some data need to be resampled, generally using the highest resolution as the reference standard, as higher resolution is more conducive to landslide extraction; in addition, landslide vector labels and auxiliary task vector labels are converted from vectors to raster images with the same resolution as other data; finally, image registration is performed to ensure that multi-source data and corresponding label data are spatially aligned.

[0016] Furthermore, in S3, the multi-task deep learning model BFM-UNet includes:

[0017] The branch fusion encoder extracts features from different source data through multiple branches, typically including branches for optical remote sensing images and topographic data. More branches can be added as needed to process other data containing landslide features, such as SAR images. Richer multi-source data helps to expand feature representation and enable the model to cope more effectively with complex environments and variable geographical features. The features extracted by each branch are fused through cross-stitch units and then passed to the decoder for further processing.

[0018] The multi-task decoder assists landslide extraction by introducing other land cover recognition tasks in addition to landslide extraction, enabling the model to more effectively utilize background feature information. At the same time, it enhances the model's feature mining capabilities through multi-task learning, reduces dependence on data, and utilizes cross-stitch unit learning to learn the optimal sharing between tasks, thereby better leveraging the advantages of multi-task learning.

[0019] Furthermore, in S3, both the branch fusion encoder and the multi-task decoder contain cross-stitch units. The original cross-stitch units were mainly used for adaptive sharing between different tasks. Here, their functional application is expanded to be used for multi-source data feature fusion. In order to meet the flexibility required for multi-source data feature fusion, the cross-stitch units are expanded, no longer restricting the number of input and output features, allowing them to be arbitrary values ​​and not necessarily the same. In addition, considering the differences between different channels of the feature map, the cross-stitch units are also improved. The weight elements in the original weight matrix are expanded from scalars to vectors with the same dimension as the number of channels of the feature map. The processing of the feature map is calculated according to formula (1):

[0020]

[0021] in, F represents the i-th output feature map. j j = 1, 2, ..., m represents the j-th input feature map; w ij This indicates that when calculating the i-th output feature map, the j-th input feature map corresponds to a weight, which is a vector with the same dimension as the number of channels in the input feature map. That is, each channel in the feature map corresponds to an element in the corresponding weight vector. The calculation of the i-th output feature map is shown in formula (2):

[0022]

[0023] The operation * is performed on a channel-by-channel basis, i.e., F j Each element in the feature map of the k-th channel is related to the vector w ij Multiply the corresponding k-th element.

[0024] Furthermore, in S3, the branch fusion encoder adopts a multi-branch structure, extracting features from different source data separately. Each branch backbone consists of one input block and four downsampling blocks. Compared to the original UNet, residual blocks replace the convolutional layers in the downsampling blocks. The input and output of the input block are defined as follows: and The height and width of the output feature map are the same as those of the input image, both being H. in and W in The number of channels is C data Convert to C in C data C represents the number of channels for the input data. in The parameter is adjustable and is usually set to 32 or 64. The processing procedure of the input block is expressed as formula (3):

[0025] Y in =ReLU(BN(Conv) 3×3 (ReLU(BN(Conv 3×3 (X in (3)

[0026] Where ReLU is the activation function, BN represents batch normalization, and Conv... 3×3 This represents a 3×3 convolution operation; the processing of the feature map by the downsampling block is expressed by formula (4):

[0027] Y d =Res(Res(MaxPool(X) d (4)

[0028] in, and H represents the input and output feature maps of the downsampling block, respectively. d ×W d ×C d Given the shape of the input feature map, Res is the residual block, and MaxPool is a max pooling operation with a window size of 2×2 and a stride of 2. After processing by the downsampling block, the height and width of the feature map are reduced by half, while the number of channels is doubled. The processing of the feature map by the residual block is expressed by formula (5):

[0029] Y res =ReLU(BN(Conv) 3×3 (ReLU(BN(Conv 3×3 (X res )))))+X res (5)

[0030] Among them, X res and Y res These represent the feature maps of the input and output, respectively.

[0031] Cross-stitch units are applied at jump connections and at the bottom connections of the encoder and decoder to learn the correlation between multi-source data, so as to fuse features more effectively and pass the fusion result to the decoder. At these locations, the number of input features and the number of output features of the cross-stitch unit correspond to the number of feature extraction branches in the encoder and the number of task branches in the decoder, respectively. The initialization method of the weight vector in the weight matrix is ​​as shown in formula (6):

[0032]

[0033] Where N represents the number of branches in the encoder and C is the dimension of the weight vector, this setting ensures that the features extracted by each branch have the same weight initially and are fused equally.

[0034] Furthermore, in S3, the multi-task decoder extends the original UNet decoder with additional branches to identify suitable non-landslide features. Multi-task learning allows the model to make fuller use of background information. Each branch consists of four upsampling blocks and one output block. Compared to the original UNet, residual blocks replace the convolutional layers in the upsampling blocks. The output blocks of the auxiliary task branches are only enabled during model training. Each upsampling block has two inputs: a feature map from either the last downsampling block in the encoder or the upsampling block from the previous layer. and feature maps from the corresponding layers of the encoder via skip connections. The output obtained after processing by the upsampling block is Where H u ×W u ×C u This represents the shape of the feature map from the bottommost downsampled block or the upsampled block of the previous layer. The processing procedure for the upsampled block is as shown in formula (7):

[0035] Y u =Res(Res(Cat(X) u2, Conv T (X u1 )))) (7)

[0036] Where Res represents the residual block, calculated according to formula (5), Cat represents the feature map by channel dimension, and Conv T This represents a transposed convolution; the output block consists of a 1×1 convolution, and its input feature map has a shape of H. out ×W out ×C in The height and width of the output remain unchanged, consistent with the height and width of the input image of the model, and the number of channels is 2, corresponding to the number of classification categories in each branch;

[0037] Each branch adds a cross-stitch unit after each layer of upsampling block for feature sharing. This unit learns the best sharing between tasks, further leveraging the advantages of the multi-task structure. The number of its input and output features corresponds to the number of task branches. When initializing the parameters of this unit, the elements of the weight vector on the diagonal of the weight matrix are all 1, and the elements of the weight vector off the diagonal are all 0, so that each task branch is independent of each other initially, as shown in formula (8).

[0038]

[0039] Among them, 1 C This represents a vector of dimension C with all elements being 1, 0, ... C This represents a vector of dimension C with all elements being 0.

[0040] Furthermore, in S4, each task corresponds to a joint loss function L, which combines a weighted cross-entropy loss L. CE And Dice lost L Dice L is calculated according to formula (9):

[0041] L = L CE +L Dice (9)

[0042] Weighted cross-entropy loss is an extension of cross-entropy loss, introducing weights to address class imbalance. Its calculation formula is as follows:

[0043]

[0044] Where B represents the total number of samples in a batch, C represents the total number of categories, and w j It is the weight of category j, y ij y is the one-hot encoded value of the true label of sample i in category j. If sample i is in category j, then y ij It is 1 if it is true, otherwise it is 0. This represents the probability that sample i belongs to class j, as predicted by the model. The class weights are set according to the degree of class imbalance, using a common method based on inverse frequencies. The calculation formula is as follows:

[0045]

[0046] Where, N j This represents the total number of samples in class j; the Dice loss is defined based on the Dice coefficient, and it optimizes model performance by maximizing the Dice coefficient. Its calculation formula is:

[0047]

[0048] Where, p ij and gij This indicates whether sample i belongs to category j in the predicted result and the true label, respectively. If it belongs, the value is 1, otherwise it is 0.

[0049] The above loss calculation is for a single task. When training for multi-task learning, learnable, dynamically changing task loss weights are used to learn the relative importance of different tasks from the data. The total loss L for multi-task learning is... MT The calculation formula is:

[0050]

[0051] Where T represents the total number of tasks, L i The loss for task i is calculated using formula (9), s i It is a learnable parameter, based on which learnable dynamic loss weights are implemented.

[0052] Further, in step S4, a dataset is constructed based on the data obtained in step S1 and processed according to step S2. This dataset is divided into a training set and a test set. Data augmentation is performed on the training set using rotation, vertical flipping, and horizontal flipping. The training set is used for training the BFM-UNet model, and the test set is used to evaluate the performance of the trained model. During model training, an appropriate batch size and training epochs are set. SGD is selected as the optimization algorithm, and a dynamically changing learning rate is used. The learning rate changes in two stages:

[0053] In the first stage, the learning rate is gradually increased linearly from a very low value to the set value during the initial training phase to reduce the impact of the model's sensitivity to the initial weights. Specifically, it is calculated according to formula (14):

[0054]

[0055] Where α is an adjustable parameter, i is the current iteration number in the first stage, and num... phase1 It is the total number of iterations in this stage, lr init The initial learning rate;

[0056] In the second stage, the learning rate gradually decays to near 0 in a non-linear manner, which helps the model to converge stably. The specific learning rate is calculated according to formula (15):

[0057]

[0058] Where β is an adjustable parameter, j is the current iteration number in the second stage, and num phase2 This represents the total number of iterations in this phase.

[0059] Furthermore, in step S5, the multi-source data of the target area obtained through step S1 and processed according to step S2 is input into the trained multi-task deep learning model BFM-UNet for landslide extraction, thereby obtaining the corresponding landslide distribution map. In the distribution map, landslide areas are displayed in white, and non-landslide areas are displayed in black.

[0060] The beneficial effects of this invention are as follows:

[0061] (1) The multi-task decoder of the present invention uses multi-task learning to add the task of identifying non-landslide features that are widely present in the target area and are easier to identify than landslides to assist in landslide extraction, so that the model can make more effective use of background information and thus improve the accuracy of landslide extraction.

[0062] (2) The multi-task decoder of the present invention utilizes multi-task learning to enhance the feature mining capability of the network, enabling the model to mine more valuable features with limited samples, improve data utilization, and effectively address the problem of small sample learning.

[0063] (3) The branch fusion encoder of the present invention extracts features from multi-source data through different branches and uses cross-stitch units to learn the correlation between data, effectively extracting and fusing feature information in multi-source data, ensuring that the model can fully utilize the information contained in multi-source data when performing landslide extraction.

[0064] (4) This invention further improves the landslide extraction accuracy by working in collaboration between the branch fusion encoder and the multi-task decoder, which can better distinguish landslides from other land features, thereby reducing the false detection rate and improving the reliability of landslide extraction results.

[0065] (5) The landslide extraction method proposed in this invention has a framework that can be adjusted according to specific task requirements, and has good adaptability and scalability. It can provide effective technical support for landslide monitoring, disaster early warning and disaster prevention and mitigation in complex geological environments.

[0066] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0067] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0068] Figure 1 This is the technical approach of the present invention;

[0069] Figure 2 It uses the BFM-UNet architecture;

[0070] Figure 3 The results of landslide extraction are compared with the actual labels. Detailed Implementation

[0071] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0072] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0073] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0074] Technical routes such as Figure 1As shown. First, the required data is acquired. Optical remote sensing imagery and DEM data serve as the basis for landslide extraction. Additionally, landslide labels and auxiliary task labels are used for model training. The auxiliary task selects widely distributed and identified land features within the target area as targets, such as vegetation, water bodies, and buildings; vegetation is used as an example here. Next, the acquired data is preprocessed. Optical remote sensing imagery requires atmospheric correction to obtain images closer to the actual surface reflection, and the RGB three-channel data is exported. Topographic analysis is performed based on DEM data to obtain topographic factors (slope and aspect); relevant label data needs to be converted from vector to raster. The processed data are then unified in resolution and image registered to ensure pixel-level correspondence within the same coordinate system. Finally, the processed optical remote sensing imagery (RGB) and topographic data (elevation, slope, and aspect) are used as input to the BFM-UNet model, and the model is trained using the processed relevant label data. Finally, by inputting the processed optical remote sensing imagery (RGB) and topographic data (elevation, slope, and aspect) into the trained model, a landslide distribution map of the corresponding area can be obtained.

[0075] In this embodiment, the BFM-Unet network architecture is as follows: Figure 2 As shown, the branch-fusion encoder employs a dual-branch structure, extracting features from 3-channel optical remote sensing imagery (Red, Green, Blue) and 3-channel topographic data (elevation, slope, aspect) respectively. These features are then fused using cross-stitch units, which learn the relationships between multi-source data, represented as corresponding weights. In the multi-task decoder, a task of identifying other land cover features (vegetation selected here) besides landslide extraction is added to assist landslide extraction. The optimal sharing between tasks is learned through cross-stitch units, also represented as corresponding weights.

[0076] (1) Branch Fusion Encoder

[0077] The encoder of the model adopts a dual-branch structure, which extracts features from 3-channel optical remote sensing images (Red, Green, Blue) and 3-channel terrain data (elevation, slope, aspect) respectively. Then, the features of the two branches are fused through the cross-stitch unit, and the fused result is passed to the decoder. Figure 1 The left side shows the specific structure of the encoder, where the main branches of the encoder each consist of one input block and four downsampling blocks. The input block mainly consists of two convolutional layers, and the input and output of the input block are defined as follows: and H in and W in These are the height and width of the input image, respectively. The height and width of the output feature map are the same as the input image, and the number of channels is converted to 32. The processing of the input block can be expressed as formula (1):

[0078] Y in =ReLU(BN(Conv) 3×3 (ReLU(BN(Conv 3×3 (X in (1)

[0079] Where ReLU is the activation function, BN represents batch normalization, and Conv... 3×3 This represents a 3×3 convolution operation. The downsampling block contains a max pooling operation and two consecutive residual blocks, and its processing of the feature map can be expressed as formula (2):

[0080] Y d =Res(Res(MaxPool(X) d (2)

[0081] in, and H represents the input and output feature maps of the downsampling block, respectively. d ×W d ×C d Given the shape of the input feature map, Res is the residual block, and MaxPool is a 2×2 window with a stride of 2, which halves the height and width of the feature map. After downsampling, the height and width of the feature map are reduced by half, while the number of channels doubles. The residual block's processing of the feature map can be expressed by formula (3):

[0082] Y res =ReLU(BN(Conv) 3×3 (ReLU(BN(Conv 3×3 (X res )))))+X res (3)

[0083] Among them, X res and Y res These represent the input and output feature maps, respectively. Furthermore, cross-stitch units are applied at skip connections and at the bottom connections of the encoder and decoder to learn the correlations between multi-source data, thus fusing features more effectively. At these locations, the number of input and output features of the cross-stitch unit corresponds to the number of feature extraction branches in the encoder and the number of task branches in the decoder, respectively, both of which are 2. For the cross-stitch unit in the encoder, the initialization method of the weight vector in the weight matrix is ​​as shown in formula (4):

[0084]

[0085] Here, N represents the number of branches in the encoder, C is the dimension of the weight vector, and i = 1, 2, j = 1, 2, N = 2. This setting ensures that the features extracted by each branch have the same weight initially, allowing for equal fusion.

[0086] (2) Multi-task decoder

[0087] Based on the UNet infrastructure, additional branches are extended in the decoder to identify suitable non-landslide features, and the best sharing between tasks is learned by using cross-stitch units, thereby constructing a multi-task network for landslide extraction, enabling the model to make fuller use of information in the background (non-landslide areas). Figure 1 The right side shows the structure of the decoder, with two branches corresponding to the extraction tasks of landslide and non-landslide features (vegetation as an example here). The module within the red dashed box is only used during model training and does not need to be enabled in actual landslide extraction applications. Each branch consists of four upsampling blocks and one output block. Each upsampling block has two inputs: a feature map from the bottom downsampling block or the upsampling block from the previous layer. and feature maps from the corresponding layers of the encoder via skip connections. The output obtained after processing by the upsampling block is Where H u ×W u ×C u This represents the shape of the feature map from the bottommost downsampled block or the upsampled block of the previous layer. The processing procedure for the upsampled block is as shown in formula (5):

[0088] Y u =Res(Res(Cat(X) u2 Conv T (X u1 (5)

[0089] Where Res represents the residual block, and the processing method is shown in formula (3), Cat represents the feature map by channel dimension, and Conv T This represents a transposed convolution. After transposed convolution, the height and width of the feature map are doubled, while the number of channels is halved. The output block of each branch in the decoder consists of a 1×1 convolution, and its input feature map has a shape of H. out ×W out×32, the height and width of the output remain unchanged, consistent with the height and width of the input image of the model, the number of channels is 2, corresponding to the number of classification categories of each branch, and each branch is a binary classification task. In the decoder, in addition to utilizing the implicit mutual influence between different task branches in the multi-task network structure, a cross-stitch unit is added after each layer upsampling block of each branch for feature sharing, so that different task branches can explicitly influence each other, and the cross-stitch unit can learn the best sharing between tasks, further leveraging the advantages of the multi-task structure. The role of the cross-stitch unit here is to realize adaptive feature sharing between different task branches, so the number of its input and output features corresponds to the number of task branches, which is 2. When initializing the parameters of this unit, the elements of the weight vector on the diagonal of the weight matrix are all 1, and the elements of the weight vector off the diagonal are all 0, so that each task branch is independent of each other at the beginning, as shown in formula (6):

[0090]

[0091] Among them, 1 C This represents a vector of dimension C with all elements being 1, 0, ... C Let i represent a vector of dimension C with all elements equal to 0, where i = 1, 2 and j = 1, 2.

[0092] (3) Loss Function

[0093] For each task, a joint loss function L is used, which combines the weighted cross-entropy loss Li. CE And Dice lost L Dice L is calculated according to formula (7):

[0094] L = L CE +L Dice (7)

[0095] Weighted cross-entropy loss is an extension of cross-entropy loss, introducing weights to address class imbalance. Its calculation formula is as follows:

[0096]

[0097] Where B represents the total number of samples in a batch (referring to pixels in the image), C represents the total number of categories, and w j It is the weight of category j, y ij y is the one-hot encoded value of the true label of sample i in category j. If sample i is in category j, then y ij It is 1 if it is true, otherwise it is 0. This represents the probability that sample i belongs to class j, as predicted by the model. The class weights are set according to the degree of class imbalance, using a common method based on inverse frequencies. The calculation formula is as follows:

[0098]

[0099] Where, N j This represents the total number of samples in class j. The Dice loss is defined based on the Dice coefficient. The Dice loss optimizes model performance by maximizing the Dice coefficient, and its calculation formula is:

[0100]

[0101] Where, p ij and g ij This indicates whether sample i belongs to category j in the predicted result and the true label, respectively. If it does, the value is 1; otherwise, it is 0.

[0102] The above loss calculation is for a single task. In multi-task learning, learnable, dynamically changing task loss weights are used to learn the relative importance of different tasks from the data. Under this method, the total loss L for multi-task learning is... MT The calculation formula is:

[0103]

[0104] Where T represents the total number of tasks, L i The loss for task i is calculated using formula (7), s i It is a learnable parameter, based on which learnable dynamic loss weights are implemented.

[0105] (4) Hyperparameter settings

[0106] During model training, the batch size was set to 4, the training epochs were 300, SGD was selected as the optimization algorithm, and a dynamically changing learning rate was used. The learning rate was changed in two stages. In the first stage, the learning rate was gradually increased linearly from a very low value to the set learning rate (0.01) in the early stage of training to reduce the influence of the model's sensitivity to the initial weights. Specifically, it was calculated according to formula (12):

[0107]

[0108] Where α is 0.001, i is the current iteration number in the first stage (starting from 1), and num... phase1 It is the total number of iterations in this stage, lr init The initial learning rate is set to 0.01. In the second stage, the learning rate gradually decays non-linearly to near 0, which helps the model converge stably. The specific learning rate is calculated according to formula (13):

[0109]

[0110] Where β is 0.9, j is the current iteration number in the second stage (starting from 1), and num phase2 This represents the total number of iterations in this phase; the first training round is the first phase, and the remaining training rounds constitute the second phase, with each training round containing 199 iterations.

[0111] A dataset was constructed based on existing labeled data, and the training set and test set were divided in a 7:3 ratio. The training set was augmented by rotation, vertical flip, and horizontal flip. The training set was used to train the BFM-UNet model.

[0112] (5) Landslide extraction

[0113] The three-channel optical remote sensing imagery (Red, Green, Blue) and three-channel topographic data (elevation, slope, aspect) of the target area are input into the trained multi-task deep learning model BFM-UNet to extract landslides, thereby obtaining the corresponding landslide distribution map. Landslide areas are displayed in white and non-landslide areas are displayed in black in the distribution map.

[0114] In this embodiment, the above method is applied to landslide extraction in the Jiuzhaigou area. Figure 3 The results of this method and the corresponding real labels are shown in several examples, and it can be seen that the two are very close.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A landslide extraction method based on multi-task learning and multi-source data feature fusion, characterized in that: Includes the following steps: S1: Data acquisition, including relevant multi-source data and vector label data, where the vector label data includes landslide labels and labels related to auxiliary tasks; S2: Preprocess the acquired data, including removing noise and interference from relevant multi-source data, converting relevant label data from vector to raster, and finally ensuring that all data have the same resolution and are spatially aligned. S3: Construct a multi-task deep learning model, BFM-UNet, for landslide extraction using multi-source data. This model includes a branch-fusion encoder and a multi-task decoder. The multi-task deep learning model BFM-UNet includes: The branch fusion encoder extracts features from different source data through multiple branches, including branches for optical remote sensing images and terrain data. More branches can be added as needed to process SAR images. The features extracted from each branch are fused through cross-stitch units and then passed to the decoder for processing. The multi-task decoder introduces other land cover recognition tasks besides landslide extraction, utilizes new feature information in the background to assist landslide extraction, enhances the model's feature mining capabilities, and learns the optimal sharing between tasks using cross-stitch units. S4: Training the model, including calculating the loss function and setting the relevant hyperparameters. During training, a joint loss function is used for each task, combining weighted cross-entropy loss and Dice loss. In multi-task learning, learnable and dynamically changing task loss weights are used. In addition, an appropriate batch size and training epochs are set, SGD is selected as the optimization algorithm, and a dynamically changing learning rate is used. S5: Based on the optical remote sensing images and topographic data of the target area after S2 preprocessing, landslides are extracted using the trained BFM-UNet model to obtain a landslide distribution map.

2. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: In step S1, the multi-source data to be acquired includes optical remote sensing images and digital elevation models, with SAR images added as needed. Landslide vector labels and auxiliary task vector labels also need to be acquired, with the auxiliary task selecting non-landslide features within the area as targets.

3. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: In step S2, the preprocessing of multi-source data includes: performing atmospheric correction on optical remote sensing images, and selecting to export all or only some channels of data as needed; performing terrain analysis based on DEM data to obtain slope and aspect data; if SAR images are available, performing radiometric correction, denoising, and terrain correction; to ensure the resolution uniformity of multi-source data, using the highest resolution data as the standard, resampling data with lower resolutions; converting landslide vector labels and auxiliary task vector labels from vectors to raster images with the same resolution as other data; and finally, performing image registration to ensure that the multi-source data and corresponding label data are spatially aligned.

4. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: Both the branch fusion encoder and the multi-task decoder contain cross-stitch units. The original cross-stitch units are used for adaptive sharing between different tasks. In the branch fusion encoder, they are used for multi-source data feature fusion. To meet the flexibility required for multi-source data feature fusion, the cross-stitch units are extended, without limiting the number of input and output features, allowing them to be arbitrary values ​​and not necessarily the same. The weight elements in the original weight matrix are expanded from scalars to vectors with the same dimension as the number of channels in the feature map. The processing of the feature map is calculated according to formula (1): （1） in, Let i = 1, 2, ..., n represent the i-th output feature map. , This represents the j-th input feature map; This indicates that when calculating the i-th output feature map, the j-th input feature map corresponds to a weight, which is a vector with the same dimension as the number of channels in the input feature map. That is, each channel in the feature map corresponds to an element in the corresponding weight vector. The calculation of the i-th output feature map is shown in formula (2): （2） Among them, operations It is done through channels, that is Each element in the feature map of the k-th channel is related to the vector. Multiply the corresponding k-th element.

5. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: The branch fusion encoder employs a multi-branch structure, extracting features from different source data separately. Each branch backbone consists of one input block and four downsampling blocks. Compared to the original UNet, residual blocks replace the convolutional layers in the downsampling blocks. The input and output of the input block are defined as follows: and The height and width of the output feature map are the same as those of the input image, both being [value missing]. and The number of channels is Convert to , The number of channels for the input data. The parameter is adjustable and can be set to 32 or 64. The processing procedure of the input block is expressed as formula (3): （3） Where ReLU is the activation function and BN represents batch normalization. express Convolution operation; the processing of the feature map by the downsampling block is expressed by formula (4): （4） in, and These represent the input and output feature maps of the downsampling block, respectively. The shape of the input feature map is defined by Res, where Res is the residual block and MaxPool is a max pooling operation with a window size of 2×2 and a stride of 2. After processing by the downsampling block, the height and width of the feature map are reduced by half, while the number of channels is doubled. The processing of the feature map by the residual block is expressed by formula (5): （5） in, and These represent the feature maps of the input and output, respectively. Cross-stitch units are applied at the jump connections and at the bottom connections of the encoder and decoder to learn the correlation between multi-source data, fuse features, and pass the fusion result to the decoder; at these locations, the number of input features and the number of output features of the cross-stitch unit correspond to the number of feature extraction branches in the encoder and the number of task branches in the decoder, respectively, and the initialization method of the weight vector in the weight matrix is ​​as shown in formula (6): （6） in, N Indicates the number of branches in the encoder. C The dimension of the weight vector is set in this way so that the features extracted by each branch have the same weight initially and are fused equally.

6. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: The multi-task decoder extends the original UNet decoder with additional branches to identify suitable non-landslide features. It utilizes multi-task learning to enable the model to make fuller use of background information. Each branch consists of four upsampling blocks and one output block. Compared to the original UNet, residual blocks replace the convolutional layers in the upsampling blocks. The output blocks of the auxiliary task branches are only enabled during model training. Each upsampling block has two inputs: a feature map from either the last downsampling block in the encoder or the upsampling block from the previous layer. and feature maps from the corresponding layers of the encoder via skip connections. The output obtained after processing by the upsampling block is ,in This represents the shape of the feature map from the bottommost downsampled block or the upsampled block of the previous layer. The processing procedure for the upsampled block is as shown in formula (7): （7） Where Res represents the residual block, calculated according to formula (5), and Cat represents the feature map by channel dimension. This represents a transposed convolution; the output block consists of a 1×1 convolution, and its input feature map has the following shape. The height and width of the output remain unchanged, consistent with the height and width of the input image of the model, and the number of channels is 2, corresponding to the number of classification categories in each branch; A cross-stitch unit is added after each layer of upsampling block in each branch for feature sharing. This unit learns the best sharing between tasks, and the number of its input and output features corresponds to the number of task branches. When initializing the parameters of this unit, the elements of the weight vector on the diagonal of the weight matrix are all 1, and the elements of the weight vector off the diagonal are all 0, so that each task branch is independent of each other at the beginning, as shown in formula (8): （8） in, This represents a vector of dimension C with all elements being 1. This represents a vector of dimension C with all elements being 0.

7. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: In S4, each task corresponds to a joint loss function L, which combines weighted cross-entropy loss. and Dice loss L is calculated according to formula (9): （9） Weighted cross-entropy loss is an extension of cross-entropy loss, introducing weights to address class imbalance. Its calculation formula is as follows: （10） Where B represents the total number of samples in a batch, and C represents the total number of categories. It is the weight of category j. It is the one-hot encoded value of the true label of sample i in category j. If sample i is in category j, then It is 1 if it is 1, otherwise it is 0. This represents the probability that sample i belongs to class j, as predicted by the model. The class weights are set according to the degree of class imbalance, using a common method based on inverse frequencies. The calculation formula is as follows: （11） in, This represents the total number of samples in class j; the Dice loss is defined based on the Dice coefficient, and it optimizes model performance by maximizing the Dice coefficient. Its calculation formula is: （12） in, and This indicates whether sample i belongs to category j in the predicted result and the true label, respectively. If it belongs, the value is 1, otherwise it is 0. The above loss calculation is for a single task. When training for multi-task learning, learnable, dynamically changing task loss weights are used to learn the relative importance of different tasks from the data, resulting in a total loss for multi-task learning. The calculation formula is: （13） Where T represents the total number of tasks. The loss for task i is calculated using formula (9). It is a learnable parameter, based on which learnable dynamic loss weights are implemented.

8. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: In step S4, a dataset is constructed based on the data obtained in S1 and processed according to S2. This dataset is divided into a training set and a test set. Data augmentation is performed on the training set using rotation, vertical flipping, and horizontal flipping. The training set is used for training the BFM-UNet model, and the test set is used to evaluate the performance of the trained model. During model training, an appropriate batch size and training epochs are set. SGD is selected as the optimization algorithm, and a dynamically changing learning rate is used. The learning rate changes in two stages: In the first stage, the learning rate is gradually increased linearly from a very low value to the set value during the initial training phase to reduce the impact of the model's sensitivity to the initial weights. Specifically, it is calculated according to formula (14): （14） in, This is an adjustable parameter, where i is the current iteration number in the first phase. This is the total number of iterations in this phase. The initial learning rate; In the second stage, the learning rate gradually decays non-linearly to near 0, allowing the model to converge stably. The specific learning rate is calculated according to formula (15): （15） in, Here, j is an adjustable parameter, representing the current iteration number in the second phase. This represents the total number of iterations in this phase.

9. The landslide extraction method based on multi-task learning and multi-source data feature fusion according to claim 1, characterized in that: In step S5, the multi-source data of the target area obtained through S1 and processed according to S2 is input into the trained multi-task deep learning model BFM-UNet for landslide extraction, thereby obtaining the corresponding landslide distribution map. In the distribution map, landslide areas are displayed in white and non-landslide areas are displayed in black.

Images