Multi-scale extraction method of large-scale landslides based on Transformer

Abstract

The present disclosure provides a Transformer-based large-scale landslide multi-scale extraction method, device, equipment, storage medium and program product, which can be applied to the fields of image processing technology and deep learning technology. The Transformer-based large-scale landslide multi-scale extraction method includes: based on the image to be extracted in a preset area and the historical image of the preset area, extracting the target pixel points related to the landslide in the image to be extracted to obtain an initial landslide image; inputting the initial landslide image into a landslide extraction model to obtain a target landslide image; wherein the landslide extraction model includes an encoder, an attention module and a decoder, the encoder is used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module is used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder is used to decode the encoded result image based on the attention feature to obtain the target landslide image.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing technology and deep learning technology, and in particular to a Transformer-based large-scale landslide multi-scale extraction method, device, equipment, medium and program product. Background Art

[0002] Landslides are a frequent geological disaster that poses a potential risk to surface infrastructure and land use. Therefore, timely and accurate acquisition of landslide information is crucial for the formulation of rescue strategies and emergency measures. In addition, detailed landslide extraction over a large spatial range can provide a comprehensive and systematic landslide catalog, which is of great significance in geological hazard risk management, landslide-related research, and planning and decision-making.

[0003] In the prior art, landslide extraction is usually performed on remote sensing images through landslide extraction models. However, the existing landslide extraction models are mostly targeted at large-scale landslides, and the extraction effect of small-scale landslides is not ideal. In addition, due to the large-scale research, there are a large number of background objects in remote sensing images, and the number of landslides and background objects in remote sensing images is seriously unbalanced, resulting in unsatisfactory effect of landslide extraction in large-scale applications. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a Transformer-based large-scale landslide multi-scale method, device, equipment, medium and program product.

[0005] According to a first aspect of the present disclosure, a Transformer-based large-scale landslide multi-scale extraction method is provided, comprising: based on an image to be extracted in a preset area and a historical image of the preset area, extracting target pixel points related to the landslide in the image to be extracted to obtain an initial landslide image; inputting the initial landslide image into a landslide extraction model to obtain a target landslide image; wherein the landslide extraction model comprises an encoder, an attention module and a decoder, the encoder being used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module being used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder being used to decode the encoded result image based on the attention feature to obtain the target landslide image.

[0006] According to an embodiment of the present disclosure, the encoder includes a plurality of serially connected encoding layers, the output of each of the encoding layers is used as the input of the next encoding layer, the feature map output by the last encoding layer is the result feature map, and the feature maps output by the encoding layers other than the last encoding layer are the intermediate feature maps; the encoder includes a SwinTransformer; the initial landslide image is input into a landslide extraction model to obtain a target landslide image, comprising: inputting the initial landslide image into the encoder for encoding to obtain the result feature map; inputting the initial landslide image and the target intermediate feature map in the at least one intermediate feature map into the attention module to obtain the attention feature; inputting the result feature map and the attention feature into the decoder to obtain the target landslide image.

[0007] According to an embodiment of the present disclosure, the above-mentioned image to be extracted based on a preset area and the historical image of the preset area, extracts target pixel points related to the landslide in the above-mentioned image to be extracted to obtain an initial landslide image, including: for each first pixel point in the above-mentioned image to be extracted, based on the second pixel point related to the above-mentioned first pixel point in the above-mentioned historical image, determining the landslide data of the above-mentioned first pixel point, wherein the above-mentioned first pixel point and the above-mentioned second pixel point correspond to the same ground point; based on a preset threshold and the landslide data of each of the above-mentioned first pixel points, screening the above-mentioned multiple first pixel points to obtain the target pixel point; and obtaining the above-mentioned initial landslide image based on the above-mentioned target pixel point.

[0008] According to an embodiment of the present disclosure, the above landslide data includes band variation, slope, snow cover index and vegetation index.

[0009] For each first pixel point in the above-mentioned image to be extracted, based on the second pixel point related to the above-mentioned first pixel point in the above-mentioned historical image, the landslide data of the above-mentioned first pixel point is determined, including: based on the band data of the above-mentioned first pixel point and the band data of the above-mentioned second pixel point, determining the band change of the band data of the above-mentioned first pixel point relative to the above-mentioned second pixel point; based on the height value of the above-mentioned first pixel point and the height values ​​of other first pixel points adjacent to the above-mentioned first pixel point, determining the slope of the above-mentioned first pixel point; and based on the band data of the above-mentioned first pixel point, obtaining the above-mentioned normalized snow accumulation index and the above-mentioned normalized vegetation index, wherein the above-mentioned snow accumulation index is used to characterize the degree of snow accumulation at the ground point corresponding to the above-mentioned first pixel point, and the above-mentioned vegetation index is used to characterize the degree of vegetation coverage at the ground point corresponding to the above-mentioned second pixel point.

[0010] According to an embodiment of the present disclosure, the above-mentioned preset thresholds include a slope threshold, a snow accumulation index threshold, a vegetation index threshold and a change amount threshold. The above-mentioned multiple first pixel points are screened based on the preset thresholds and the landslide data of each of the above-mentioned first pixel points to obtain target pixel points, including: comparing the slope data of the above-mentioned first pixel point with the above-mentioned slope threshold to obtain a first comparison result; comparing the snow accumulation index of the above-mentioned first pixel point with the above-mentioned snow accumulation index threshold to obtain a second comparison result; comparing the vegetation index of the above-mentioned first pixel point with the above-mentioned vegetation index threshold to obtain a third comparison result; comparing the band change amount of the above-mentioned first pixel point with the above-mentioned change amount threshold to obtain a fourth comparison result; and determining the target pixel point among the above-mentioned multiple first pixel points based on the first comparison result, the second comparison result, the third comparison result and the fourth comparison result of each of the above-mentioned multiple first pixel points, wherein the ground point corresponding to the above-mentioned target pixel point has the possibility of being a landslide.

[0011] According to an embodiment of the present disclosure, the above-mentioned initial landslide image and the target intermediate feature map in the above-mentioned at least one intermediate feature map are input into the above-mentioned attention module to obtain the above-mentioned attention feature, including: based on multiple preset scales, feature extraction is performed on the above-mentioned initial landslide image to obtain a first feature map corresponding to each of the above-mentioned preset scales; based on the above-mentioned multiple preset scales, feature extraction is performed on the above-mentioned target intermediate feature map to obtain a second feature map corresponding to each of the above-mentioned preset scales; and for each of the above-mentioned preset scales, based on the first feature map corresponding to the above-mentioned preset scale and the second feature map corresponding to the above-mentioned preset scale, an attention feature corresponding to the above-mentioned preset scale is obtained.

[0012] According to an embodiment of the present disclosure, the above-mentioned feature extraction is performed on the above-mentioned initial landslide image based on multiple preset scales to obtain a first feature map corresponding to each of the above-mentioned preset scales, including: convolving the above-mentioned initial landslide image to obtain a first convolution feature; dividing the above-mentioned first convolution feature into multiple feature blocks according to multiple preset scales to obtain a first feature block set corresponding to each of the above-mentioned preset scales; determining the first feature vector of each of the multiple first feature blocks in the above-mentioned first feature block set, wherein the above-mentioned first feature vector is a one-dimensional vector; and arranging the multiple first feature vectors according to a preset rule to obtain the above-mentioned first feature map corresponding to each of the above-mentioned preset scales.

[0013] According to an embodiment of the present disclosure, based on the above-mentioned multiple preset scales, feature extraction is performed on the above-mentioned target intermediate feature map to obtain a second feature map corresponding to each of the above-mentioned preset scales: convolution is performed on the above-mentioned target intermediate feature map to obtain a second convolution feature; according to the above-mentioned multiple preset scales, the above-mentioned second convolution feature is divided into multiple feature blocks to obtain a second feature block set corresponding to each of the above-mentioned preset scales; the second feature vector of each of the multiple second feature blocks in the above-mentioned second feature block set is determined, wherein the above-mentioned second feature vector is a one-dimensional vector; and the multiple second feature vectors are arranged according to a preset rule to obtain the above-mentioned second feature map corresponding to each of the above-mentioned preset scales.

[0014] According to an embodiment of the present disclosure, the above-mentioned second feature map includes a first channel feature related to the landslide and a second channel feature related to the background objects; for each of the above-mentioned preset scales, based on the first feature map corresponding to the above-mentioned preset scale and the second feature map corresponding to the above-mentioned preset scale, the attention feature corresponding to the above-mentioned preset scale is obtained, including: multiplying the above-mentioned first feature map with the transposed first channel feature and the above-mentioned second channel feature respectively, to obtain a first sub-attention feature related to the landslide and a second sub-attention feature related to the background objects.

[0015] According to an embodiment of the present disclosure, the above-mentioned result feature map and the above-mentioned attention feature are input into the above-mentioned decoder to obtain the above-mentioned target landslide image, including: based on multiple preset scales, feature extraction is performed on the above-mentioned result feature map to obtain a third feature map corresponding to each of the above-mentioned preset scales, wherein the above-mentioned third feature map includes a third channel feature related to the landslide and a fourth channel feature related to the background object; for each of the above-mentioned preset scales, the third channel feature is multiplied by the above-mentioned first sub-attention feature to obtain a fifth channel feature; the fourth channel feature is multiplied by the above-mentioned second sub-attention feature to obtain a sixth channel feature; the fifth channel feature is fused with the sixth channel feature to obtain a fourth feature map; according to the fourth feature maps of each of the above-mentioned multiple preset scales, a fifth feature map fused with multiple preset scales is obtained; and the fifth feature map is decoded to obtain the above-mentioned target landslide image.

[0016] A second aspect of the present disclosure provides a Transformer-based large-scale landslide multi-scale extraction device, comprising: a first extraction module, used to extract target pixel points related to the landslide in the image to be extracted based on the image to be extracted in a preset area and the historical image of the preset area, so as to obtain an initial landslide image; a second extraction module, used to input the initial landslide image into a landslide extraction model, so as to obtain a target landslide image; wherein the landslide extraction model comprises an encoder, an attention module and a decoder, the encoder is used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module is used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder is used to decode the encoded result image based on the attention feature to obtain the target landslide image.

[0017] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the Transformer-based large-scale landslide multi-scale extraction method.

[0018] The fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned Transformer-based large-scale landslide multi-scale extraction method.

[0019] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned Transformer-based large-scale landslide multi-scale extraction method.

[0020] The Transformer-based large-scale landslide multi-scale extraction method provided by the present disclosure includes at least the following technical effects:

[0021] (1) A potential landslide extraction method is proposed to reduce the impact of a large number of background objects on landslide extraction, so as to deal with the problem of data imbalance in large-scale landslide detection, thereby enabling the landslide extraction model to be effectively applied in a large range of complex backgrounds.

[0022] (2) The cross-feature map attention is used to improve the Swin Transformer to supplement the information loss during feature map propagation and make up for the feature loss of small landslides, thereby improving the model's ability to extract small landslides.

[0023] (3) Construct a multi-scale cross-feature map attention module so that the attention module takes into account both cross-feature map and multi-scale information and retains landslide information of different scales to achieve multi-scale extraction of landslides and improve the model's extraction results of landslides of different scales in a large range. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0025] Figure 1 A schematic diagram showing an application scenario of a Transformer-based large-scale landslide multi-scale extraction method, device, equipment, medium, and program product according to an embodiment of the present disclosure;

[0026] Figure 2 The flowchart of the Transformer-based large-scale landslide multi-scale extraction method according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 3 A schematic diagram of extracting potential landslides according to a specific embodiment of the present disclosure is schematically shown;

[0028] Figure 4 A model structure diagram of a landslide extraction model according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 5 The structure diagram of the attention module according to a specific embodiment of the present disclosure is schematically shown;

[0030] Figure 6 The landslide extraction result according to a specific embodiment of the present disclosure is schematically shown;

[0031] Figure 7 Schematically showing the landslide extraction result according to another specific embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of a structure block diagram of a large-scale landslide multi-scale extraction device based on Transformer according to an embodiment of the present disclosure is shown; and

[0033] Fig. 9 A block diagram of an electronic device suitable for implementing a Transformer-based large-scale landslide multi-scale extraction method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0035] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0036] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0037] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0038] When extracting landslides from remote sensing, image processing methods are usually used in combination with computer vision and machine learning technologies to build an automatic landslide extraction model. Although both pixel-based and object-based semi-automatic landslide extraction methods have been widely used in landslide extraction, both methods require manual threshold setting. In comparison, machine learning can automatically learn feature thresholds through training samples and has obvious advantages in application effects. However, machine learning is usually limited to manually selected features, which restricts its ability to achieve ideal accuracy in complex environments. In order to overcome this challenge, deep learning has become an important technology in the field of landslide extraction in recent years. It learns features and patterns from a large amount of data and adapts more flexibly to different geological and geomorphological conditions, thereby improving the accuracy and robustness of landslide extraction. Yi et al. constructed the LandsNet deep learning network and further optimized the extraction results obtained through morphology, achieving an accuracy of 86.89% in the study area.

[0039] Existing landslide extraction methods are mostly targeted at large-scale landslides, and generally have unsatisfactory extraction effects on small-scale landslides. And because the research is mostly limited to a single or a few events, the research scope is small, resulting in a relatively simple background object learned by the landslide extraction model, which is difficult to be effectively applied in a large range. The main reason for these problems is that the information of small-scale landslides is easily lost during the feature propagation of deep learning, resulting in the omission of small-scale landslides. In addition, the existing landslide extraction models mostly rely on remote sensing images with a balanced number of landslides and background objects, but due to the large-scale research, a large number of background objects are brought to the remote sensing images, resulting in a serious imbalance in the number of landslides and background objects in the remote sensing images, resulting in unsatisfactory application of landslide extraction in a large range. The present disclosure retains the small-scale landslide information in the image by introducing multi-scale cross-feature attention in the deep learning model, and before the remote sensing image is input into the deep learning model, a large number of background objects in the remote sensing image that are obviously not landslides are eliminated to achieve a relative balance between the number of landslides and the number of background objects, so that the deep learning model can focus on distinguishing the background objects that are easily confused with landslides, and improve the ability to extract landslides.

[0040] An embodiment of the present disclosure provides a Transformer-based large-scale landslide multi-scale extraction method, comprising: based on an image to be extracted in a preset area and a historical image of the preset area, extracting target pixel points related to the landslide in the image to be extracted to obtain an initial landslide image; inputting the initial landslide image into a landslide extraction model to obtain a target landslide image; wherein the landslide extraction model comprises an encoder, an attention module and a decoder, the encoder is used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module is used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder is used to decode the encoded result image based on the attention feature to obtain the target landslide image.

[0041] Figure 1 The application scenario diagram of the Transformer-based large-scale landslide multi-scale extraction method according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, and a third terminal device 103. A network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and a server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0043] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0044] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the Transformer-based large-scale landslide multi-scale extraction method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the Transformer-based large-scale landslide multi-scale extraction method device provided in the embodiment of the present disclosure can generally be set in the server 105. The Transformer-based large-scale landslide multi-scale extraction method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the Transformer-based large-scale landslide multi-scale extraction method device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] The following will be based on Figure 1 The scene described by Figure 2 to Figure 7The Transformer-based large-scale landslide multi-scale extraction method of the disclosed embodiment is described in detail.

[0049] Figure 2 The flowchart of the Transformer-based large-scale landslide multi-scale extraction method according to an embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the Transformer-based large-scale landslide multi-scale extraction method of this embodiment includes operations S210 to S220.

[0051] In operation S210, based on the image to be extracted in the preset area and the historical image of the preset area, target pixel points related to the landslide in the image to be extracted are extracted to obtain an initial landslide image.

[0052] In operation S220, the initial landslide image is input into a landslide extraction model to obtain a target landslide image.

[0053] According to an embodiment of the present disclosure, the preset area may be a large area where landslide extraction is required. The image to be extracted and the historical image may be synthesized by satellite data of the preset area. Specifically, remote sensing data of multiple satellites such as Landsat-5, Landsat-7 and Landsat-8 may be obtained through a global scale remote sensing cloud computing platform (Google Earth Engine, GEE).

[0054] According to an embodiment of the present disclosure, before extracting target pixel points related to the landslide in the image to be extracted based on the image to be extracted in the preset area and the historical image of the preset area to obtain the initial landslide image, it also includes: creating the image to be extracted in the preset area and the historical image of the preset area.

[0055] According to an embodiment of the present disclosure, for each pixel in the image to be extracted, the Normalized Difference Vegetation Index (NDVI) of the pixel at multiple time points within the time period to be extracted can be determined using satellite data, and the time point corresponding to the median of multiple NDVI values ​​is determined as the synthetic image time of the pixel. The synthetic image to be extracted is generated using the satellite data corresponding to the synthetic image time of each pixel. The synthetic image to be extracted is visually interpreted to obtain the image to be extracted. Specifically, the time period to be extracted can be one year, and the obtained synthetic image to be extracted is the annual synthetic image to be extracted.

[0056] According to an embodiment of the present disclosure, the method for creating a historical image may be the same as the method for creating an image to be extracted, and will not be described in detail herein. In the case where the composite image to be extracted is an annual composite image to be extracted, the historical composite image may be an annual historical composite image of the year before the time period to be extracted. For example, the composite image to be extracted may be a composite image of a preset area in the current year, and the historical composite image may be a composite image of the preset area in the historical year before the current year. Accordingly, the image to be extracted is the image data of the preset area in the current year, and the historical image is the image data of the preset area in the historical year before the current year.

[0057] According to an embodiment of the present disclosure, the target pixel point may be a pixel point related to a landslide in the image to be extracted. Specifically, the initial landslide image may be obtained by comparing the image to be extracted with the historical image, screening the target pixel point where a potential landslide exists at the ground point corresponding to the pixel point among the multiple pixel points of the image to be extracted. The target pixel point may also be determined directly based on the attribute data of the pixel point itself in the image to be extracted.

[0058] According to the embodiments of the present disclosure, by screening the target pixel points in the image to be extracted, a large number of background objects in the image to be extracted that are obviously unrelated to the landslide are effectively eliminated, so that the number of landslides and the number of background objects in the initial landslide image are relatively balanced.

[0059] According to an embodiment of the present disclosure, based on the image to be extracted in a preset area and the historical image of the preset area, target pixel points related to the landslide in the image to be extracted are extracted. Before obtaining the initial landslide image, the image to be extracted and the historical image can also be adjusted to an image of 256×256 pixels.

[0060] According to an embodiment of the present disclosure, a landslide extraction model includes an encoder, an attention module and a decoder. The encoder is used to encode an initial landslide image to obtain at least one intermediate feature map and a result feature map. The attention module is used to obtain attention features based on the initial landslide image and the intermediate feature map. The decoder is used to decode the encoded result image based on the attention features to obtain a target landslide image.

[0061] According to an embodiment of the present disclosure, by introducing an attention module in the encoder-decoder structure, the attention module can supplement information for the intermediate feature map through the initial landslide image, compensate for the information loss of small-scale landslides during the feature map propagation process, and better retain the landslide information.

[0062] According to the embodiments of the present disclosure, cross-feature attention is introduced into the deep learning model to retain small-scale landslide information in the image. Before the remote sensing image is input into the deep learning model, a large number of background objects in the remote sensing image that are obviously not landslides are eliminated to achieve a relative balance between the number of landslides and the number of background objects. This allows the deep learning model to focus on distinguishing background objects that are easily confused with landslides, thereby improving the landslide extraction capability.

[0063] According to an embodiment of the present disclosure, for each first pixel point in the image to be extracted, the landslide data of the first pixel point is determined based on the second pixel point related to the first pixel point in the historical image; based on a preset threshold and the landslide data of each first pixel point, multiple first pixel points are screened to obtain a target pixel point; and based on the target pixel point, an initial landslide image is obtained.

[0064] According to an embodiment of the present disclosure, the first pixel point may be a pixel point in the image to be extracted, and the second pixel point may be a pixel point in the historical image. The landslide data may be used to determine whether the ground point corresponding to the first pixel point is a potential landslide. Specifically, the landslide data of the first pixel point may be determined by comparing the difference between the first pixel point and the second pixel point corresponding to the same ground point, and the landslide data of the first pixel point may also be directly determined by using the attribute data of the first pixel point.

[0065] According to an embodiment of the present disclosure, the preset threshold may be a threshold of landslide data. When the landslide data is screened using the preset threshold, the first pixel point of the landslide data within the preset threshold range may be determined as the target pixel point.

[0066] According to an embodiment of the present disclosure, when an initial landslide image is obtained based on a target pixel point, a zero-filling process may be performed on the first pixel point except the target pixel point, and no limitation is imposed on this.

[0067] According to the embodiments of the present disclosure, in order to solve the data imbalance problem caused by the significant difference in the number of landslides and background objects in a large area, the target pixel points are screened out from the first pixel points by a preset threshold value to obtain an initial landslide image. A large number of background objects that are obviously not landslides are effectively eliminated, and the data is relatively balanced, so that the landslide extraction model is more focused on distinguishing landslides from the background objects that are most easily confused with them, thereby improving the landslide extraction capability.

[0068] According to an embodiment of the present disclosure, based on the band data of the first pixel point and the band data of the second pixel point, the band change of the band data of the first pixel point relative to the second pixel point is determined; based on the height value of the first pixel point and the height values ​​of other first pixel points adjacent to the first pixel point, the slope of the first pixel point is determined; and based on the band data of the first pixel point, a normalized snow index and a normalized vegetation index are obtained.

[0069] According to an embodiment of the present disclosure, landslide data includes band variation, slope, snow accumulation index and vegetation index. The band variation is used to characterize the band variation of the band data of the first pixel point relative to the band data of the second pixel point, the snow accumulation index is used to characterize the degree of snow accumulation at the ground point corresponding to the first pixel point, and the vegetation index is used to characterize the degree of vegetation coverage at the ground point corresponding to the second pixel point.

[0070] According to an embodiment of the present disclosure, the band change amount of the band data of the first pixel point relative to the band data of the second pixel point can be determined by using a change vector analysis method. Specifically, the band data of the first pixel point and the band data of the second pixel point can be respectively converted into vectors with a certain direction and intensity, and the change vector intensity between the two vectors is determined as the band change amount.

[0071] According to the embodiments of the present disclosure, the band data of the first pixel point can characterize the surface state of the ground point corresponding to the first pixel point to a certain extent, and by determining the band change amount of the band data of the first pixel point relative to the band data of the second pixel point, it can be determined whether the surface state of the ground point has changed significantly. In the case where the band change amount is large, it can be considered that the ground point is more likely to be a landslide; in the case where the band change amount is small, it can be considered that the ground point is less likely to be a landslide.

[0072] According to an embodiment of the present disclosure, the height value of the first pixel point can be obtained through a digital elevation model (DEM). When determining the slope of the first pixel point, the slope can be determined by the height value of the first pixel point whose slope is to be determined and the height values ​​of other first pixel points adjacent to the first pixel point whose slope is to be determined.

[0073] According to the embodiments of the present disclosure, when the slope is small, it can be considered that the ground point is relatively flat. Since landslides do not occur in flat areas, when the slope is small, it can be considered that the ground point is less likely to be a landslide.

[0074] According to an embodiment of the present disclosure, the normalized snow index may be the Normalized Difference Snow Index (NDSI), and the normalized vegetation index may be the Normalized Difference Vegetation Index (NDVI).

[0075] According to an embodiment of the present disclosure, the NDSI can be calculated using the band data of the first pixel. Specifically, the NDSI can be calculated using the following formula (1):

[0076] NDSI=(GREEN-SWIR) / (GREEN+SWIR) (1)

[0077] Among them, GREEN is the green band data of the first pixel point, and SWIR is the shortwave infrared band data of the first pixel point.

[0078] According to an embodiment of the present disclosure, NDVI can also be calculated using the band data of the first pixel. Specifically, NDVI can be calculated using the following formula (2):

[0079] NDVI=(NIR-RED) / (NIR+RED) (2)

[0080] Wherein, NIR is the reflectivity of the first pixel point in the near infrared band, and RED is the reflectivity of the first pixel point in the red band.

[0081] According to the embodiments of the present disclosure, when the NDSI is large, it can be considered that the snow accumulation level at the ground point is high and the possibility of a landslide at the ground point is low; when the NDSI is small, it can be considered that the snow accumulation level at the ground point is low and the possibility of a landslide at the ground point is high.

[0082] According to the embodiments of the present disclosure, when the NDVI is small, it can be considered that the vegetation coverage of the ground point is low and the possibility of a landslide at the ground point is high; when the NDVI is large, it can be considered that the vegetation coverage of the ground point is high and the possibility of a landslide at the ground point is low.

[0083] According to the embodiments of the present disclosure, by determining the band variation, slope, snow accumulation index and vegetation index of the first pixel point, the accuracy of landslide data is improved, and the balance between the number of landslides and the number of background objects in the initial landslide data is further achieved.

[0084] According to an embodiment of the present disclosure, based on a preset threshold and the landslide data of each first pixel point, multiple first pixel points are screened to obtain a target pixel point, including: comparing the slope data of the first pixel point with the slope threshold to obtain a first comparison result; comparing the normalized snow index of the first pixel point with the snow index threshold to obtain a second comparison result; comparing the normalized vegetation index of the first pixel point with the vegetation index threshold to obtain a third comparison result; comparing the band change of the first pixel point with the change threshold to obtain a fourth comparison result; and determining the target pixel point among the multiple first pixel points based on the first comparison result, the second comparison result, the third comparison result and the fourth comparison result of each of the multiple first pixel points.

[0085] According to an embodiment of the present disclosure, the preset thresholds include a slope threshold, a snow index threshold, a vegetation index threshold and a change threshold. The slope threshold is used to compare with the slope data, the snow index threshold is used to compare with the normalized snow index, the vegetation index threshold is used to compare with the normalized vegetation index, and the change threshold is used to compare with the band change.

[0086] According to an embodiment of the present disclosure, the ground point corresponding to the target pixel point may be a landslide.

[0087] According to an embodiment of the present disclosure, when the first comparison result represents that the slope data is greater than the slope threshold, the first pixel point is determined to be the target pixel point; when the second comparison result represents that the normalized snow index is less than the snow index threshold, the first pixel point is determined to be the target pixel point; when the third comparison result represents that the normalized vegetation index is less than the vegetation index threshold, the first pixel point is determined to be the target pixel point; when the fourth comparison result represents that the band change is greater than the change threshold, the first pixel point is determined to be the target pixel point.

[0088] According to the embodiments of the present disclosure, multiple thresholds are set to respectively judge the band variation, slope, snow index and vegetation index, thereby improving the accuracy of the target pixel point.

[0089] Figure 3 The figure schematically shows a schematic diagram of extracting potential landslides according to a specific embodiment of the present disclosure.

[0090] According to an embodiment of the present disclosure, based on the band data and height data of each first pixel point in the image to be extracted 10, the slope 301, snow index 302 and vegetation index 303 of each first pixel point in the image to be extracted 10 can be obtained.

[0091] According to an embodiment of the present disclosure, a change vector analysis is performed on a first pixel point of the image to be extracted 10 and a second pixel point of the historical image 20 to obtain a change vector diagram 304 with the same scale as the image to be extracted, wherein the band data of each pixel point in the change vector diagram 304 represents the amount of change in the band data of the first pixel point relative to the band data of the second pixel point.

[0092] According to an embodiment of the present disclosure, by performing threshold segmentation on the slope 301, snow accumulation index 302 and vegetation index 303 of each first pixel point and the band data change in the change vector diagram 304, an initial landslide image 30 is obtained based on the target pixel points among multiple first pixel points.

[0093] According to an embodiment of the present disclosure, an initial landslide image is input into a landslide extraction model to obtain a target landslide image, including: inputting the initial landslide image into an encoder for encoding to obtain a result feature map; inputting the initial landslide image and a target intermediate feature map in at least one intermediate feature map into an attention module to obtain an attention feature; inputting the result feature map and the attention feature into a decoder to obtain a target landslide image.

[0094] According to an embodiment of the present disclosure, the encoder includes multiple serially connected encoding layers, the output of each encoding layer is used as the input of the next encoding layer, the feature map output by the last encoding layer is the result feature map, and the feature maps output by other encoding layers except the last encoding layer are intermediate feature maps.

[0095] According to an embodiment of the present disclosure, the encoder includes a Swin Transformer.

[0096] According to an embodiment of the present disclosure, the target intermediate feature map may be one of the intermediate feature maps. Specifically, the target intermediate feature map may be an intermediate feature map output by the first encoding layer.

[0097] According to an embodiment of the present disclosure, the attention module realizes attention across feature maps through the initial landslide image and the target intermediate feature map, thereby better retaining the landslide information.

[0098] Figure 4 The model structure diagram of the landslide extraction model according to the embodiment of the present disclosure is schematically shown.

[0099] like Figure 4 As shown, the landslide extraction model includes an encoder 401, an attention module 402 and a decoder 403. The initial landslide image 30 is input into the encoder 401 to obtain a target intermediate feature map 41 and a result feature map 42. The initial landslide image 30 and the target intermediate feature map 41 are input into the attention module 402, and the attention features and the result feature map 42 obtained are input into the decoder 403 to obtain the target landslide image 40.

[0100] According to an embodiment of the present disclosure, an initial landslide image and a target intermediate feature map in at least one intermediate feature map are input into an attention module to obtain an attention feature, including: based on multiple preset scales, feature extraction is performed on the initial landslide image to obtain a first feature map corresponding to each preset scale; based on multiple preset scales, feature extraction is performed on the target intermediate feature map to obtain a second feature map corresponding to each preset scale; and for each preset scale, based on the first feature map corresponding to the preset scale and the second feature map corresponding to the preset scale, an attention feature corresponding to the preset scale is obtained.

[0101] According to an embodiment of the present disclosure, the preset scale may be a scale for cutting the initial landslide image and the target intermediate feature map. Each preset scale corresponds to a first feature map and a second feature map. Specifically, the initial landslide image and the target intermediate feature map may be cut into a plurality of first sub-feature maps and second sub-feature maps of a preset scale, respectively, and the first feature map is obtained according to the plurality of first sub-feature maps, and the second feature map is obtained according to the plurality of second sub-feature maps.

[0102] According to an embodiment of the present disclosure, the first feature map and the second feature map corresponding to the same preset scale as the first feature map are fused to obtain an attention feature.

[0103] According to the embodiments of the present disclosure, multi-scale cross-feature map attention is used to take into account the extraction of multi-scale landslide information, compensate for the information loss of landslides of different scales during the feature map propagation process, and better retain the landslide information.

[0104] According to an embodiment of the present disclosure, based on multiple preset scales, feature extraction is performed on an initial landslide image to obtain a first feature map corresponding to each preset scale, including: convolving the initial landslide image to obtain a first convolution feature; dividing the first convolution feature into multiple feature blocks according to the multiple preset scales to obtain a first feature block set corresponding to each preset scale; determining a first feature vector of each of the multiple first feature blocks in the first feature block set; and arranging the multiple first feature vectors according to a preset rule to obtain a first feature map corresponding to each preset scale.

[0105] According to an embodiment of the present disclosure, the first convolution feature is obtained by performing a convolution operation on the initial landslide image. When dividing the first convolution feature, the first convolution feature can be split according to multiple preset scales, for example, when the first convolution feature is split using the first preset scale, multiple first feature blocks of the same size as the first preset scale are obtained, and when the first convolution feature is split using the second preset scale, multiple first feature blocks of the same size as the second preset scale are obtained.

[0106] According to an embodiment of the present disclosure, the first eigenvector is a one-dimensional vector. Specifically, the first feature block is a two-dimensional matrix, and the first feature block can be expanded to obtain a one-dimensional first eigenvector. For example, when the dimension of the first feature block is 3×3, the dimension of the first eigenvector is 1×9.

[0107] According to an embodiment of the present disclosure, multiple first feature vectors may be arranged and concatenated according to a preset rule to obtain a first feature map. Specifically, the arrangement order of the first feature vectors may be determined according to the position of the first feature block in the first convolution feature.

[0108] According to an embodiment of the present disclosure, each preset scale corresponds to a first feature map, that is, the number of the first feature maps is equal to the preset scale.

[0109] According to the embodiments of the present disclosure, multiple first feature maps are obtained according to multiple preset scales, so that each of the obtained first feature maps corresponds to information of different scales, taking into account the extraction of multi-scale landslide information, compensating for the information loss of landslides of different scales in the process of feature map propagation, and better retaining landslide information.

[0110] According to an embodiment of the present disclosure, based on multiple preset scales, feature extraction is performed on a target intermediate feature map to obtain a second feature map corresponding to each preset scale, including: convolving the target intermediate feature map to obtain a second convolution feature; dividing the second convolution feature into multiple feature blocks according to multiple preset scales to obtain a second feature block set corresponding to each preset scale; determining a second feature vector of each of the multiple second feature blocks in the second feature block set, wherein the second feature vector is a one-dimensional vector; and arranging the multiple second feature vectors according to a preset rule to obtain a second feature map corresponding to each preset scale.

[0111] According to an embodiment of the present disclosure, the second convolution feature is obtained by performing a convolution operation on the target intermediate feature map. Specifically, when performing a convolution operation on the target intermediate feature map, the target intermediate feature map can be converted into features of multiple different categories, such as a landslide category and a background category.

[0112] According to an embodiment of the present disclosure, the preset scale for dividing the second convolution feature is the same as the preset scale for dividing the first convolution. When dividing the second convolution feature, the second convolution feature can be split according to multiple preset scales, for example, when the first convolution feature is split using the first preset scale, multiple second feature blocks of the same size as the first preset scale are obtained, and when the second convolution feature is split using the second preset scale, multiple second feature blocks of the same size as the second preset scale are obtained.

[0113] According to an embodiment of the present disclosure, the second eigenvector is a one-dimensional vector. Specifically, the second feature block is a three-dimensional matrix, and the second feature block can be expanded to obtain a one-dimensional second eigenvector. For example, when the dimension of the second feature block is 3×3, the dimension of the second eigenvector is 1×9.

[0114] According to an embodiment of the present disclosure, multiple second feature vectors may be arranged and concatenated according to a preset rule to obtain a second feature map. Specifically, the arrangement order of the second feature vectors may be determined according to the position of the second feature block in the second convolution feature.

[0115] According to an embodiment of the present disclosure, each preset scale corresponds to a second feature map, that is, the number of the second feature maps is equal to the number of the second feature maps.

[0116] According to an embodiment of the present disclosure, multiple second feature maps are obtained according to multiple preset scales, so that each obtained second feature map corresponds to information of different scales, taking into account the extraction of multi-scale landslide information and better retaining the landslide information.

[0117] According to an embodiment of the present disclosure, for each preset scale, based on a first feature map corresponding to the preset scale and a second feature map corresponding to the preset scale, an attention feature corresponding to the preset scale is obtained, including: multiplying the first feature map with the transposed first channel feature and the second channel feature respectively to obtain a first sub-attention feature related to the landslide and a second sub-attention feature related to the background object.

[0118] According to an embodiment of the present disclosure, the second feature map may include a first channel feature related to the landslide and a second channel feature related to the background object. The second channel feature related to the background object may be a feature of the background object. Specifically, the target intermediate feature map may be convolved with a 1×1 convolution layer having two filters to obtain a second convolution feature, the second convolution feature including the feature of the first channel related to the landslide and the feature of the second channel related to the background object.

[0119] According to an embodiment of the present disclosure, when splitting the second convolution feature, the features of the first channel related to the landslide and the features of the second channel related to the background objects can be respectively split to obtain two sub-feature block sets. After the sub-feature blocks are expanded into second feature vectors, the arrangement order of the second feature vectors is determined according to the positions of the sub-feature blocks in the second convolution feature, and a second feature map including the first channel features related to the landslide and the second channel features related to the background objects is obtained.

[0120] According to an embodiment of the present disclosure, by multiplying the first feature map with the transposed first channel feature and the second channel feature respectively, a first sub-attention feature related to the landslide and a second sub-attention feature related to the background feature are obtained. Specifically, the first feature map is multiplied with the transposed first channel feature to obtain the first sub-attention feature related to the landslide, and the first feature map is multiplied with the transposed second channel feature to obtain the second sub-attention feature related to the landslide.

[0121] According to an embodiment of the present disclosure, an attention feature can be obtained by splicing the first sub-attention feature and the second sub-attention feature. That is, each preset scale corresponds to an attention feature, and each attention feature includes a first sub-attention feature related to the landslide and a second sub-attention feature related to the background object.

[0122] According to an embodiment of the present disclosure, by multiplying different channel features of the first feature map and the second feature map respectively, the relationship between the initial landslide image and the second feature maps of different categories can be determined.

[0123] Figure 5 The structural diagram of the attention module according to a specific embodiment of the present disclosure is schematically shown.

[0124] like Figure 5 As shown, the input image with a dimension of H×W×3 is convolved to obtain a first convolution feature with a dimension of H×W×1, wherein the input image is the initial landslide image, H is the height of the input image and the first convolution feature, W is the width of the input image and the first convolution feature, 3 is the number of channels of the input image, and 1 is the number of channels of the first convolution feature. According to the first preset scale d 1 , the second preset scale d 2 , the third preset scale d 3 , ..., the i-th preset scale d i After the first convolution feature is divided into blocks, the first preset scale d 1 The corresponding first feature map and the second preset scale d 2 The corresponding first feature map and the third preset scale d 3 The first feature map corresponding to the i-th preset scale d i The corresponding first feature map.

[0125] According to the i-th preset scale d i The dimension of the first feature map obtained after dividing the first convolution feature into blocks is in, The number of rows that can represent the first feature map, d i 2 The number of first pixels in each row of the first feature map can be represented.

[0126] According to an embodiment of the present disclosure, for a dimension H 1 ×W 1 ×C 1 Convolution is performed on the intermediate feature map to obtain a dimension of H×W×N c The second convolutional feature of , where the intermediate feature map can be the target feature map, H 1 is the height of the intermediate feature map, W 1 is the width of the intermediate feature map, C 1 is the number of channels of the intermediate feature map, N c is the number of channels of the second convolution feature. According to the first preset scale d 1 , the second preset scale d 2 , the third preset scale d 3, ..., the i-th preset scale d i After dividing the second convolution feature into blocks, we get the same scale as the first preset scale d 1 The corresponding second feature map and the second preset scale d 2 The corresponding second feature map and the third preset scale d 3 The corresponding second feature map, and so on, until the i-th preset scale d is obtained i The corresponding second feature map.

[0127] According to the i-th preset scale d i The dimension of the second feature map obtained after dividing the second convolutional feature into blocks is in, The number of rows that can represent the second feature map, d i 2 It can represent the number of second pixels in each row of the second feature map, N c is the number of channels of the second feature map.

[0128] According to an embodiment of the present disclosure, a matrix multiplication operation is performed on the first feature map and the second feature map of the same preset scale to obtain a matrix multiplication operation with the first preset scale d 1 The corresponding attention feature MAFMA (Multi-scale Across Feature Map Attention)-1, and the second preset scale d 2 The corresponding attention feature MAFMA-2, and the third preset scale d 3 The corresponding attention feature MAFMA-3, and so on, until the i-th preset scale d is obtained i The corresponding attention feature MAFMA-i.

[0129] With the i-th preset scale d i The corresponding dimension of the attention feature is in, is the height of the attention feature, is the width of the attention feature, N c is the number of channels of the attention feature.

[0130] According to an embodiment of the present disclosure, the result feature map and the attention feature are input into a decoder to obtain a target landslide image, including: based on multiple preset scales, feature extraction is performed on the result feature map to obtain a third feature map corresponding to each preset scale; for each preset scale, the third channel feature and the first sub-attention feature are multiplied to obtain a fifth channel feature; the fourth channel feature and the second sub-attention feature are multiplied to obtain a sixth channel feature; the fifth channel feature and the sixth channel feature are fused to obtain a fourth feature map; according to the fourth feature maps of each of the multiple preset scales, a fifth feature map fused with the multiple preset scales is obtained; and the fifth feature map is decoded to obtain the target landslide image.

[0131] According to an embodiment of the present disclosure, the structure of the decoder can be symmetrical with that of the encoder. Similarly, in order to supplement the information in the attention feature into the result feature map, it is necessary to convert the result feature map into a third feature map corresponding to the preset scale according to multiple preset scales.

[0132] According to an embodiment of the present disclosure, the third feature map includes a third channel feature related to the landslide and a fourth channel feature related to the background object. Specifically, when performing a matrix multiplication operation on the third feature map and the attention feature, the third channel feature related to the landslide and the first channel feature can be multiplied to obtain a fifth channel feature, and the fourth channel feature related to the background object and the second channel feature can be multiplied to obtain a sixth channel feature.

[0133] According to an embodiment of the present disclosure, each preset scale corresponds to a fourth feature map, and each fourth feature map includes a fifth channel feature related to the landslide and a sixth channel feature related to the background object.

[0134] According to an embodiment of the present disclosure, when a plurality of fourth feature maps of different preset scales are fused into a fifth feature map, a concatenation operation can be performed, and a decoder is used to decode the fused fifth feature map to obtain a target landslide image.

[0135] According to the embodiments of the present disclosure, attention features are used to compensate for the information loss of landslides of different sizes during the feature map propagation process, so as to better retain the landslide information.

[0136] According to an embodiment of the present disclosure, when training the landslide extraction model, Stochastic Gradient Descent (SGD) with a momentum of 0.9 is used, the weight decay is 1e-3, the learning rate is set to 0.01, and the batch size is set to 8.

[0137] Figure 6 The landslide extraction result according to a specific embodiment of the present disclosure is schematically shown.

[0138] like Figure 6 As shown, Figure 6 Including (a) and (b). The landslide extraction is performed on the image to be extracted shown in (a) by using the Transformer-based large-scale landslide multi-scale extraction method disclosed in the present invention to obtain the landslide extraction result shown in (b).

[0139] Figure 7 The landslide extraction result according to another specific embodiment of the present disclosure is schematically shown.

[0140] like Figure 7 As shown, Figure 7 Including (c) and (d). The landslide extraction is performed on the image to be extracted shown in Figure (b) by using the Transformer-based large-scale landslide multi-scale extraction method disclosed in the present invention, and the landslide extraction result shown in Figure (c) is obtained.

[0141] According to the embodiments of the present disclosure, the model is trained by the training set, and the landslide extraction result is obtained on the test set. It can be seen that the method proposed in the present disclosure can obtain relatively accurate and complete extraction results for small-scale landslides, showing its excellent potential in dealing with landslide detection in complex geographical environments. In order to quantitatively evaluate the landslide extraction performance of the present invention, precision, recall, F1 score and intersection of union (IOU) are used for evaluation, and the calculation methods of these indicators are shown in the following formulas (3)-(6).

[0142]

[0143]

[0144]

[0145]

[0146] Among them, TP represents the number of pixels correctly classified as landslides, FP represents the number of pixels misclassified as non-landslides, and FN represents the number of landslide pixels misclassified as non-landslides.

[0147] As shown in Table 1 below, compared with other methods, the method of the present disclosure reaches the highest level in all four evaluation indexes, proving that the present disclosure can obtain more ideal landslide extraction results.

[0148] Table 1

[0149] method IOU(%) Precision(%) Recall(%) F1-score(%) DeepLabv3+ 21.33 43.35 29.57 35.16 ResNet 21.94 41.26 31.91 35.99 HRNet 22.33 39.07 34.31 36.54 SwinTransformer 25.43 49.65 34.27 40.55 The Disclosure 30.79 55.22 41.04 47.09

[0150] Based on the above Transformer-based large-scale landslide multi-scale extraction method, the present disclosure also provides a Transformer-based large-scale landslide multi-scale extraction device. Figure 8 The device is described in detail.

[0151] Figure 8 The structural block diagram of a large-scale landslide multi-scale extraction device based on Transformer according to an embodiment of the present disclosure is schematically shown.

[0152] like Figure 8 As shown, the Transformer-based large-scale landslide multi-scale extraction device 800 of this embodiment includes a first extraction module 810 and a second extraction module 820 .

[0153] The first extraction module 810 is used to extract target pixels related to the landslide in the image to be extracted based on the image to be extracted in the preset area and the historical image of the preset area to obtain an initial landslide image. In one embodiment, the first extraction module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0154] The second extraction module 820 is used to input the initial landslide image into the landslide extraction model to obtain a target landslide image; wherein the landslide extraction model includes an encoder, an attention module and a decoder, the encoder is used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module is used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder is used to decode the encoded result image based on the attention feature to obtain the target landslide image. In one embodiment, the second extraction module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0155] According to an embodiment of the present disclosure, the encoder includes multiple serially connected encoding layers, the output of each encoding layer is used as the input of the next encoding layer, the feature map output by the last encoding layer is the result feature map, and the feature maps output by other encoding layers except the last encoding layer are intermediate feature maps; the encoder includes a Swin Transformer.

[0156] According to an embodiment of the present disclosure, the second extraction module 820 includes an encoding submodule, an attention submodule and a decoding submodule.

[0157] The encoding submodule is used to input the initial landslide image into the encoder for encoding to obtain the resulting feature map.

[0158] The attention submodule is used to input the initial landslide image and the target intermediate feature map in at least one intermediate feature map into the attention module to obtain the attention feature.

[0159] The decoding submodule is used to input the result feature map and attention features into the decoder to obtain the target landslide image.

[0160] According to an embodiment of the present disclosure, the first extraction submodule 810 includes a data determination submodule, a screening determination submodule and an image determination submodule.

[0161] The data determination submodule is used to determine the landslide data of each first pixel point in the image to be extracted based on a second pixel point related to the first pixel point in the historical image, wherein the first pixel point and the second pixel point correspond to the same ground point.

[0162] The screening and determination submodule is used to screen a plurality of first pixel points based on a preset threshold and the landslide data of each first pixel point to obtain a target pixel point.

[0163] The image determination submodule is used to obtain the initial landslide image based on the target pixel points.

[0164] According to an embodiment of the present disclosure, the landslide data includes band variation, slope, snow accumulation index and vegetation index.

[0165] According to an embodiment of the present disclosure, the data determination submodule includes a change determination unit, a slope determination unit, and an index determination unit.

[0166] The change determination unit is used to determine a band change amount of the band data of the first pixel point relative to the band data of the second pixel point based on the band data of the first pixel point and the band data of the second pixel point.

[0167] The slope determination unit is used to determine the slope of the first pixel point based on the height value of the first pixel point and the height values ​​of other first pixel points adjacent to the first pixel point.

[0168] The index determination unit is used to obtain a normalized snow accumulation index and a normalized vegetation index based on the band data of the first pixel point, wherein the snow accumulation index is used to characterize the degree of snow accumulation at the ground point corresponding to the first pixel point, and the vegetation index is used to characterize the degree of vegetation coverage at the ground point corresponding to the second pixel point.

[0169] According to an embodiment of the present disclosure, the screening submodule includes a first comparison unit, a second comparison unit, a third comparison unit, a fourth comparison unit and a target determination unit.

[0170] The first comparison unit is used to compare the slope data of the first pixel point with the slope threshold to obtain a first comparison result.

[0171] The second comparison unit is used to compare the snow accumulation index of the first pixel point with the snow accumulation index threshold to obtain a second comparison result.

[0172] The third comparison unit is used to compare the vegetation index of the first pixel point with the vegetation index threshold to obtain a third comparison result.

[0173] The fourth comparison unit is used to compare the band change amount of the first pixel point with the change amount threshold to obtain a fourth comparison result.

[0174] The target determination unit is used to determine a target pixel point among multiple first pixel points based on the first comparison result, the second comparison result, the third comparison result and the fourth comparison result of each of the multiple first pixel points, wherein the ground point corresponding to the target pixel point may be a landslide.

[0175] According to an embodiment of the present disclosure, the attention submodule includes a first extraction unit, a second extraction unit and an attention determination unit.

[0176] The first extraction unit is used to extract features from the initial landslide image based on a plurality of preset scales to obtain a first feature map corresponding to each preset scale.

[0177] The second extraction unit is used to perform feature extraction on the target intermediate feature map based on multiple preset scales to obtain a second feature map corresponding to each preset scale.

[0178] The attention determination unit is used to obtain, for each preset scale, an attention feature corresponding to the preset scale based on a first feature map corresponding to the preset scale and a second feature map corresponding to the preset scale.

[0179] According to an embodiment of the present disclosure, the first extraction unit includes a first convolution subunit, a first division subunit, a first determination subunit and a first arrangement subunit.

[0180] The first convolution subunit is used to convolve the initial landslide image to obtain the first convolution feature.

[0181] The first division subunit is used to divide the first convolution feature into multiple feature blocks according to multiple preset scales to obtain a first feature block set corresponding to each preset scale.

[0182] The first determining subunit is used to determine a first feature vector of each of the plurality of first feature blocks in the first feature block set, wherein the first feature vector is a one-dimensional vector.

[0183] The first arrangement subunit is used to arrange the multiple first feature vectors according to a preset rule to obtain a first feature map corresponding to each preset scale.

[0184] According to an embodiment of the present disclosure, the second extraction unit includes a second convolution subunit, a second division subunit, a second determination subunit and a second arrangement subunit.

[0185] The second convolution subunit is used to convolve the target intermediate feature map to obtain the second convolution feature.

[0186] The second division subunit is used to divide the second convolution feature into multiple feature blocks according to multiple preset scales to obtain a second feature block set corresponding to each preset scale.

[0187] The second determining subunit is used to determine the second feature vector of each of the plurality of second feature blocks in the second feature block set, wherein the second feature vector is a one-dimensional vector.

[0188] The second arrangement subunit is used to arrange the multiple second feature vectors according to a preset rule to obtain a second feature map corresponding to each preset scale.

[0189] According to an embodiment of the present disclosure, the second feature map includes a first channel feature related to the landslide and a second channel feature related to the background object.

[0190] According to an embodiment of the present disclosure, the attention determination unit includes an attention determination subunit.

[0191] The attention determination subunit is used to multiply the first feature map with the transposed first channel feature and the second channel feature respectively to obtain a first sub-attention feature related to the landslide and a second sub-attention feature related to the background object.

[0192] According to an embodiment of the present disclosure, the decoding submodule includes a third extraction unit, a first fusion unit, a second fusion unit and a decoding unit.

[0193] The third extraction unit is used to perform feature extraction on the result feature map based on multiple preset scales to obtain a third feature map corresponding to each preset scale, wherein the third feature map includes a third channel feature related to the landslide and a fourth channel feature related to the background object.

[0194] The first fusion unit is used to multiply the third channel feature and the first sub-attention feature for each preset scale to obtain the fifth channel feature; multiply the fourth channel feature and the second sub-attention feature to obtain the sixth channel feature; and fuse the fifth channel feature and the sixth channel feature to obtain the fourth feature map.

[0195] The second fusion unit is used to obtain a fifth feature map that fuses multiple preset scales according to the respective fourth feature maps of multiple preset scales.

[0196] The decoding unit is used to decode the fifth feature map to obtain a target landslide image.

[0197] According to an embodiment of the present disclosure, any multiple modules in the first extraction module 810 and the second extraction module 820 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first extraction module 810 and the second extraction module 820 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first extraction module 810 and the second extraction module 820 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0198] Fig. 9 A block diagram of an electronic device suitable for implementing a Transformer-based large-scale landslide multi-scale extraction method according to an embodiment of the present disclosure is schematically shown.

[0199] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0200] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0201] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0202] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0203] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0204] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0205] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0206] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0207] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0208] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0209] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0210] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0211] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A Transformer-based large-scale landslide multi-scale extraction method, characterized in that: The method comprises: Based on the image to be extracted in the preset area and the historical image of the preset area, target pixel points related to the landslide in the image to be extracted are extracted to obtain an initial landslide image; Inputting the initial landslide image into a landslide extraction model to obtain a target landslide image; The landslide extraction model comprises an encoder, an attention module and a decoder, wherein the encoder is used to encode the initial landslide image to obtain at least one intermediate feature map and a result feature map, the attention module is used to obtain an attention feature based on the initial landslide image and the intermediate feature map, and the decoder is used to decode the result feature map based on the attention feature to obtain the target landslide image; The encoder comprises a plurality of serially connected encoding layers, the output of each encoding layer is used as the input of the next encoding layer, the feature map output by the last encoding layer is the result feature map, and the feature maps output by the other encoding layers except the last encoding layer are the intermediate feature maps; the encoder comprises a Swin Transformer; The step of inputting the initial landslide image into a landslide extraction model to obtain a target landslide image comprises: Inputting the initial landslide image into the encoder for encoding to obtain the result feature map; Inputting the initial landslide image and the target intermediate feature map in the at least one intermediate feature map into the attention module to obtain the attention feature; Inputting the result feature map and the attention feature into the decoder to obtain the target landslide image; The step of inputting the initial landslide image and the target intermediate feature map in the at least one intermediate feature map into the attention module to obtain the attention feature comprises: Based on a plurality of preset scales, feature extraction is performed on the initial landslide image to obtain a first feature map corresponding to each of the preset scales; Based on the multiple preset scales, feature extraction is performed on the target intermediate feature map to obtain a second feature map corresponding to each of the preset scales; and For each of the preset scales, based on a first feature map corresponding to the preset scale and a second feature map corresponding to the preset scale, obtaining an attention feature corresponding to the preset scale; The method of extracting features from the initial landslide image based on a plurality of preset scales to obtain a first feature map corresponding to each preset scale includes: Convolving the initial landslide image to obtain a first convolution feature; According to a plurality of preset scales, the first convolution feature is divided into a plurality of feature blocks to obtain a first feature block set corresponding to each of the preset scales; Determine a first feature vector of each of the plurality of first feature blocks in the first feature block set, wherein the first feature vector is a one-dimensional vector; and Arrange the plurality of first feature vectors according to a preset rule to obtain the first feature map corresponding to each of the preset scales, wherein the preset rule is used to indicate that the plurality of first feature vectors are to be arranged according to a position of the first feature block in the first convolution feature.

2. The method according to claim 1, characterized in that The step of extracting target pixel points related to the landslide in the image to be extracted based on the image to be extracted in the preset area and the historical image of the preset area to obtain an initial landslide image includes: For each first pixel point in the image to be extracted, determining the landslide data of the first pixel point based on a second pixel point related to the first pixel point in the historical image, wherein the first pixel point and the second pixel point correspond to the same ground point; Based on a preset threshold and the landslide data of each of the first pixel points, screening the plurality of the first pixel points to obtain a target pixel point; and Based on the target pixel points, the initial landslide image is obtained.

3. The method according to claim 2, characterized in that The landslide data include band variation, slope, snow cover index and vegetation index. The step of determining, for each first pixel point in the image to be extracted, landslide data of the first pixel point based on a second pixel point related to the first pixel point in the historical image, comprises: Based on the band data of the first pixel point and the band data of the second pixel point, determining a band change amount of the band data of the first pixel point relative to the band data of the second pixel point; determining a slope of the first pixel based on a height value of the first pixel and height values ​​of other first pixel points adjacent to the first pixel; and Based on the band data of the first pixel point, a normalized snow accumulation index and a normalized vegetation index are obtained, wherein the snow accumulation index is used to characterize the degree of snow accumulation at the ground point corresponding to the first pixel point, and the vegetation index is used to characterize the degree of vegetation coverage at the ground point corresponding to the second pixel point.

4. The method according to claim 3, characterized in that The preset thresholds include a slope threshold, a snow index threshold, a vegetation index threshold and a change threshold. The step of screening the plurality of first pixel points based on a preset threshold and the landslide data of each first pixel point to obtain a target pixel point includes: Compare the slope of the first pixel with the slope threshold to obtain a first comparison result; Comparing the normalized snow accumulation index of the first pixel with the snow accumulation index threshold to obtain a second comparison result; Comparing the normalized vegetation index of the first pixel with the vegetation index threshold to obtain a third comparison result; Compare the wavelength variation of the first pixel with the variation threshold to obtain a fourth comparison result; and Based on the first comparison result, the second comparison result, the third comparison result and the fourth comparison result of each of the plurality of first pixel points, a target pixel point among the plurality of first pixel points is determined, wherein the ground point corresponding to the target pixel point has the possibility of landslide.

5. The method according to claim 1, characterized in that The step of extracting features from the target intermediate feature map based on the multiple preset scales to obtain a second feature map corresponding to each preset scale includes: Convolving the target intermediate feature map to obtain a second convolution feature; According to the multiple preset scales, the second convolution feature is divided into multiple feature blocks to obtain a second feature block set corresponding to each of the preset scales; Determine a second feature vector of each of the plurality of second feature blocks in the second feature block set, wherein the second feature vector is a one-dimensional vector; and Arrange the plurality of second feature vectors according to a preset rule to obtain the second feature map corresponding to each of the preset scales, wherein the preset rule is used to indicate that the plurality of second feature vectors are to be arranged according to a position of the second feature block in the second convolution feature.

6. The method according to claim 1, characterized in that The second feature map includes a first channel feature associated with the landslide and a second channel feature associated with the background object; The step of obtaining, for each of the preset scales, an attention feature corresponding to the preset scale based on a first feature map corresponding to the preset scale and a second feature map corresponding to the preset scale, comprises: The first feature map is multiplied by the transposed first channel feature and the transposed second channel feature respectively to obtain a first sub-attention feature related to the landslide and a second sub-attention feature related to the background object.

7. The method according to claim 6, characterized in that The step of inputting the result feature map and the attention feature into the decoder to obtain the target landslide image comprises: Based on a plurality of preset scales, feature extraction is performed on the result feature map to obtain a third feature map corresponding to each preset scale, wherein the third feature map includes a third channel feature related to the landslide and a fourth channel feature related to the background object; For each of the preset scales, multiplying the third channel feature and the first sub-attention feature to obtain a fifth channel feature; multiplying the fourth channel feature and the second sub-attention feature to obtain a sixth channel feature; fusing the fifth channel feature and the sixth channel feature to obtain a fourth feature map; Obtaining a fifth feature map integrating multiple preset scales according to the respective fourth feature maps of the multiple preset scales; and The fifth feature map is decoded to obtain the target landslide image.

Images