Remote sensing landslide target detection model, method, system and readable medium

Abstract

The present invention relates to the field of remote sensing image target detection technology, in particular to a remote sensing landslide target detection model, method, system and readable medium. The remote sensing landslide target detection model proposed in the present invention first pre-trains an embedding layer, a position encoding module and an attention feature extraction module on a first training set to achieve learning of knowledge attributes associated with an auxiliary image; and then further trains the attention feature extraction module and the Mask‑RCNN model on a complete data set to achieve fusion of knowledge features and visible light image features, comprehensively describe the features of the landslide, and automatically extract complex features using a deep learning model to improve the detection capability of the landslide area.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image target detection, in particular to a remote sensing landslide target detection model, method, system and readable medium. Background Art

[0002] Remote sensing landslide target detection refers to the process of automatically identifying and locating landslide areas on the earth's surface using remote sensing technology. Landslides are caused by the displacement of soil or rocks along the slope due to geological and climatic factors, which often cause serious natural disasters. Image data obtained through remote sensing technology can be used to analyze landslide areas in order to quickly and accurately detect the location and scale of landslides, providing support for disaster management and emergency response.

[0003] Although remote sensing landslide target detection technology has played an important role in disaster monitoring, there are still some shortcomings and challenges: 1. The environment where landslides occur is usually complex and changeable, including different geological conditions, vegetation coverage, meteorological changes, etc. These factors will affect the quality of remote sensing images and information extraction, making the detection of landslide targets difficult. For example, areas with dense vegetation may obscure landslide features and affect detection accuracy. 2. The landslide detection process does not fully utilize all the potential information in remote sensing data. Sometimes the detection work only relies on a single type of remote sensing data, or the data fusion and feature extraction process may not be comprehensive enough, resulting in the failure to mine useful information. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned prior art that remote sensing landslide target detection cannot fully mine useful information and has low detection accuracy, the present invention proposes a training method for a remote sensing landslide target detection model. The trained remote sensing landslide target detection model can fully extract potential information through the fusion of auxiliary images and visible light images, thereby greatly improving the landslide detection accuracy.

[0005] The present invention proposes a training method for a remote sensing landslide target detection model, comprising the following steps:

[0006] S1. Construct a data set to store landslide samples {(visible light image, auxiliary image); (mask image, bounding box)}, where the auxiliary image is a grayscale image with associated attribute features; both the visible light image and the grayscale image are captured images of the detection area; the mask image is used to mark the landslide area on the visible light image; and the bounding box is used to mark the landslide area on the visible light image;

[0007] S2. Construct a knowledge embedding model. The knowledge embedding model includes an embedding module, an attention feature extraction module, a position encoding module, and a segmentation head connected in sequence. The knowledge embedding model takes the auxiliary image as input and the mask image as output.

[0008] S3, extracting a first training set {auxiliary image, mask image} from the data set, and training the knowledge embedding model on the first training set until convergence;

[0009] S4, extracting a sequentially connected embedding module, an attention feature extraction module, and a position encoding module from the knowledge embedding model, and connecting the position encoding module to the Mask-RCNN model through a convolution module to form a basic model; the input data of the basic model includes a visible light image and an auxiliary image, and the output is a visible light image with annotated bounding boxes and mask images;

[0010] S5. Train the basic model on the data set {(visible light image, auxiliary image); (mask image, bounding box)} to update the attention feature extraction module, convolution module and Mask-RCNN model until the basic model converges; the converged basic model is the remote sensing landslide target detection model.

[0011] Preferably, the attribute characteristics include one or more of elevation, slope, aspect, plan curvature, profile curvature, vegetation coverage, annual rainfall, water flow intensity index and terrain wetness index.

[0012] Preferably, the base model is trained on the dataset, and the loss function used in the training process is: the sum of classification loss, bounding box regression loss and mask segmentation loss.

[0013] Preferably, the classification loss Loss cls The calculation formula is:

[0014]

[0015] Among them, y i is the true category label, y i =1 means a landslide is detected, y i =0 means no landslide was detected; p i represents the landslide probability predicted by the basic model; N is the number of categories, including those with landslide and those without landslide.

[0016] Preferably, the bounding box regression loss Loss box The calculation formula is:

[0017]

[0018] Where M is the number of landslide sample anchor frames; t mn Represents the basic model prediction value of the regression target n of the mth landslide sample anchor frame. The regression target includes the center coordinates (x, y) of the anchor frame, the width w' of the anchor frame, and the height h', t' of the anchor frame mn represents the true value of the regression target n of the mth landslide sample anchor box; (t mn-t' mn ) indicates t mn and t' mn ; SmoothL1 is the value selection function; x' is a proxy parameter.

[0019] Preferably, the mask segmentation loss Loss mask The calculation formula is:

[0020]

[0021] Where H is the height of the visible light image, W is the width of the visible light image; y h,w represents the value of the pixel coordinate (h, w) on the real mask image corresponding to the visible light image; p h,w It represents the probability that the pixel coordinate (h, w) value on the mask image predicted by the basic model is 1; ln represents the logarithmic function.

[0022] Preferably, the embedding module adopts query dictionary embedding; the position encoding module adopts rotation encoding, and the attention feature extraction module adopts CBAM convolutional attention module.

[0023] A remote sensing landslide target detection method proposed by the present invention comprises:

[0024] St1. Obtain remote sensing landslide target detection model; obtain visible light images and auxiliary images of the detection area;

[0025] St2. Input the visible light image and the auxiliary image into the remote sensing landslide target detection model, and the remote sensing landslide target detection model outputs the visible light image with the marked bounding box and mask image.

[0026] The invention proposes a remote sensing landslide target detection system, comprising an unmanned aerial vehicle, a memory and a processor, wherein the unmanned aerial vehicle is used to collect visible light images and auxiliary images; a computer program is stored in the memory, and the processor is connected to the memory and the unmanned aerial vehicle, and the processor is used to execute the computer program to implement the remote sensing landslide target detection method.

[0027] The present invention provides a readable medium storing a computer program, which is used to implement the remote sensing landslide target detection method when executed.

[0028] The advantages of the present invention are:

[0029] (1) The training method of the remote sensing landslide target detection model proposed in the present invention constructs a knowledge feature extractor consisting of an embedding module, a position encoding module, and an attention feature extraction module, which fully analyzes the potential features of the auxiliary image and effectively improves the accuracy of remote sensing landslide target detection and the efficiency of multivariate data utilization.

[0030] (2) The remote sensing landslide target detection model proposed in the present invention first pre-trains the embedding module, the position encoding module and the attention feature extraction module on the first training set to achieve the learning of the knowledge attributes associated with the auxiliary image; then further trains the attention feature extraction module and the Mask-RCNN model on the complete data set to achieve the fusion of knowledge features and visible light image features, so as to combine knowledge features and image features to comprehensively describe the characteristics of the landslide, and use the deep learning model to automatically extract complex features, thereby improving the detection capability of the landslide area.

[0031] (3) The remote sensing landslide target detection method proposed in the present invention adopts the above-mentioned remote sensing landslide target detection model to perform target detection under the support of multi-dimensional data, thereby improving data utilization and detection accuracy.

[0032] (4) The remote sensing landslide target detection system and readable medium proposed in the present invention provide a carrier for the remote sensing landslide target detection model and method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a training method for a remote sensing landslide target detection model;

[0034] Figure 2 It is a structural diagram of the knowledge embedding model;

[0035] Figure 3 This is the structure diagram of the remote sensing landslide target detection model;

[0036] Figure 4 It is the topological map of the remote sensing landslide target detection model;

[0037] Figure 5 This is a comparison of the average accuracy of the two models in the embodiment;

[0038] Figure 6 This is the detection result of the KGE-Mask-RCNN model. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Reference Figure 1 The training method of the remote sensing landslide target detection model proposed in this embodiment includes the following steps:

[0041] S1. Construct a data set to store landslide samples {(visible light image, auxiliary image); (mask image, bounding box)}, where the visible light image is a photographed image of the detection area; the auxiliary image is a grayscale image with associated attribute features, and the attribute features include elevation, slope, slope aspect, plane curvature, profile curvature, vegetation coverage, annual rainfall, water flow intensity index and terrain humidity index; the mask image is used to mark the landslide area on the visible light image; the bounding box is recorded in an XML file; the bounding box is used to mark the landslide area on the visible light image.

[0042] Grayscale images correlating elevation and vegetation coverage are collected by UAV remote sensing, i.e., the drone takes the image and annotates the elevation and vegetation coverage;

[0043] Annual rainfall is obtained through statistical data, and NASA's GPM data can be directly downloaded;

[0044] Slope, aspect, plan curvature, and profile curvature can be calculated by combining elevation and vegetation coverage;

[0045] The stream intensity index and topographic wetness index are obtained by combining elevation with vegetation cover and annual rainfall.

[0046] Visible light images were obtained by drone photogrammetry.

[0047] S2. Construction Figure 2 The knowledge embedding model shown in the figure includes: an embedding module, an attention feature extraction module, a position encoding module and a segmentation head connected in sequence; the knowledge embedding model takes the auxiliary image as input and the mask image as output;

[0048] The embedding module is used to convert the auxiliary image into a knowledge embedding vector. The attention feature extraction module is used to extract the attention features of the knowledge embedding vector, which are then passed to the position encoding module for feature encoding. The segmentation head generates a mask image based on the feature encoding.

[0049] Specifically, the embedding module can use a query dictionary to perform embedding operations so as to map the low-dimensional auxiliary image to a high-dimensional space to form a knowledge embedding vector. In this embodiment, the embedding module establishes a dictionary for each piece of knowledge; the vocabulary size of the query dictionary is set to 64 in the embedding module, and the embedding dimension is 4, and then each pixel is mapped to a 4N-dimensional vector through query, where N is the number of attribute features; and then a 4N channel feature map of the auxiliary image is formed by pixel splicing.

[0050] The position encoding module adopts rotation encoding, and the attention feature extraction module adopts CBAM convolutional attention module; the segmentation head consists of four layers of sequentially connected convolutions, the number of convolution kernels are set to 64, 512, 32 and 2, the sizes of convolution kernels are 7×7, 3×3, 3×3 and 1×1, the step sizes are all 1, and the padding distances are 3, 1, 1 and 0, respectively.

[0051] S3. Extract the first training set {auxiliary image, mask image} from the data set, and train the knowledge embedding model on the first training set until convergence.

[0052] The convergence conditions of the knowledge embedding model can be set as: the number of training times reaches the set value, the model accuracy converges, the model loss converges, etc.

[0053] S4. Extract the sequentially connected embedding module, attention feature extraction module and position encoding module from the knowledge embedding model, and connect the position encoding module to the Mask-RCNN model through the convolution module to form Figure 3 , Figure 4 The basic model shown in the figure; the input data of the basic model includes visible light images and auxiliary images, and the output is a visible light image with annotated bounding boxes and mask images; the auxiliary images are processed into encoding features by the embedding module, the attention feature extraction module and the position encoding module, the encoding features are convolved by the convolution module and then concatenated with the visible light image, and then input into the Mask-RCNN model for processing, and the Mask-RCNN model outputs the bounding box and mask image of the annotated landslide area.

[0054] That is, the input of the base model is connected to the input of the embedding module and the input of the Mask-RCNN model respectively, and the output of the base model is the output of the Mask-RCNN model.

[0055] In the basic model, visible light images usually include three channels: R, G, and B. The 4N-dimensional coded features output by the coded features adjust the number of channels through the convolutional model to avoid weight deviation between the number of coded feature channels and the number of visible light channels.

[0056] S5. Train the basic model on the data set {(visible light image, auxiliary image); (mask image, bounding box)} to update the attention feature extraction module, convolution module and Mask-RCNN model until the basic model converges; the converged basic model is the remote sensing landslide target detection model.

[0057] The training process of the basic model includes the following sub-steps:

[0058] S51, dividing the data set into a training set and a validation set;

[0059] S52, extracting training samples {(visible light image, auxiliary image); (mask image, bounding box)} from the training set, allowing the basic model to learn the training samples, and updating the attention feature extraction module, convolution module and Mask-RCNN model;

[0060] S53, extracting test samples {(visible light image, auxiliary image); (mask image, bounding box)} from the training set, allowing the basic model to predict the bounding box and mask image based on the visible light image and the auxiliary image, and calculating the model loss Loss according to the true value and predicted value of the bounding box and the mask image; iterating the attention feature extraction module, convolution module and Mask-RCNN model in the basic model through the model loss back propagation;

[0061] Loss=Loss box +Loss mask +Loss cls

[0062] Among them, Loss cls 、Loss box and Loss mask They represent classification loss, bounding box regression loss, and mask segmentation loss, respectively, and are calculated as follows:

[0063]

[0064] Among them, y i is the true category label, y i =1 means landslide is detected, y i =0 means no landslide was detected; p i represents the landslide probability predicted by the basic model; N = 2, represents the category, i.e., landslide and no landslide;

[0065]

[0066] Where M is the number of landslide sample anchor frames; t mn Represents the basic model prediction value of the regression target n of the mth landslide sample anchor frame. The regression target includes the coordinates (x, y) of the anchor frame center, the anchor frame width w' and the anchor frame height h'. x is the horizontal coordinate of the anchor frame center on the image, and y is the vertical coordinate of the anchor frame center on the image. t' mn represents the true value of the regression target n of the mth landslide sample anchor box; (t mn -t' mn ) indicates t mn and t' mn SmoothL1 is the value selection function; x' is a proxy parameter; the anchor box is a pre-set size, and each pixel is traversed to obtain all pixel boxes that contain the landslide target as much as possible.

[0067]

[0068] H is the height of the visible light image, W is the width of the visible light image; y h,w Indicates the value of the pixel coordinate (h, w) on the real mask image corresponding to the visible light image, y h,w = 0 means the pixel coordinate (h, w) is not in the landslide area, y h,w =1 means the pixel coordinate (h,w) is in the landslide area; p h,w Represents the probability that the pixel coordinate (h, w) value on the mask image predicted by the basic model is 1;

[0069] S54, determine whether the number of model iterations reaches the set threshold; if not, return to step S52; if yes, execute step S55;

[0070] S55. Calculate the performance index of the basic model on the validation set to determine whether the performance index has reached convergence; if not, return to step S51 and re-count the number of iterations of the basic model; if yes, fix the basic model and complete the training.

[0071] The remote sensing landslide target detection method proposed in this embodiment includes the following steps:

[0072] St1. Obtain remote sensing landslide target detection model; obtain visible light images and auxiliary images of the detection area;

[0073] Visible light images are taken by drones;

[0074] The auxiliary image is obtained by first using drone patrol to obtain images with marked elevation and vegetation coverage of the detection area; then combining NASA's GPM data set, marking the annual rainfall on the image, and then combining the elevation, vegetation coverage and annual rainfall to calculate the slope aspect, plane curvature, profile curvature, water flow intensity index and terrain wetness index, thereby obtaining a grayscale image with multiple knowledge features such as elevation, vegetation coverage, annual rainfall, calculated slope aspect, plane curvature, profile curvature, water flow intensity index and terrain wetness index as an auxiliary image;

[0075] St2. Input the visible light image and the auxiliary image into the remote sensing landslide target detection model. The auxiliary image is processed into coding features by the embedding module, the attention feature extraction module and the position encoding module in turn. The coding features are processed by the convolution module and then input into the Mask-RCNN model together with the visible light image for processing. The Mask-RCNN model outputs the visible light image with the marked bounding box and the mask image.

[0076] The above remote sensing landslide target detection model is described below in conjunction with specific embodiments.

[0077] In this embodiment, firstly, a data set {(visible light image, auxiliary image); (mask image, bounding box)} is constructed according to known landslide samples, and the data set is divided into a training set and a validation set.

[0078] In this embodiment, firstly, samples {auxiliary image, mask image} are extracted from the data set to train the knowledge embedding model until the number of training times reaches 25; then, a remote sensing landslide target detection model is trained on the training set, which is recorded as KGE-Mask-RCNN.

[0079] In this embodiment, the training set is also used to directly train the Mask-RCNN model as a comparison model.

[0080] In this embodiment, the average accuracy mPA of the KGE-Mask-RCNN model and the Mask-RCNN model is verified on the validation set. As the number of iterations (epochs) on the training set increases, the average accuracy is as follows: Figure 5 As shown. It can be seen that the average accuracy of the Mask-RCNN model converges to about 0.7, while the average accuracy of the KGE-Mask-RCNN model converges to about 0.8; it proves that the KGE-Mask-RCNN model has greatly improved the detection accuracy of landslides. After the trained KGE-Mask-RCNN model processes a sample {visible light image, auxiliary image} in the validation set, the landslide area (landslide) bounding box output by the KGE-Mask-RCNN model is as follows Figure 6 As shown, it can be seen that the annotation results are very clear and accurate. Both landslide areas on the landslide image are marked, and the confidence levels of the identification results reach 93% and 97% respectively.

[0081] Of course, it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0082] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0083] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A training method for a remote sensing landslide target detection model, characterized in that: The following steps are involved: S1. Construct a data set to store landslide samples {(visible light image, auxiliary image); (mask image, bounding box)}, where the auxiliary image is a grayscale image with associated attribute features; both the visible light image and the grayscale image are captured images of the detection area; the mask image is used to mark the landslide area on the visible light image; and the bounding box is used to mark the landslide area on the visible light image; S2. Build a knowledge embedding model. The knowledge embedding model includes an embedding module, an attention feature extraction module, a position encoding module, and a segmentation head connected in sequence. The knowledge embedding model takes the auxiliary image as input and the mask image as output; S3, extracting a first training set {auxiliary image, mask image} from the data set, and training the knowledge embedding model on the first training set until convergence; S4, extracting a sequentially connected embedding module, an attention feature extraction module, and a position encoding module from the knowledge embedding model, and connecting the position encoding module to the Mask-RCNN model through a convolution module to form a basic model; the input data of the basic model includes a visible light image and an auxiliary image, and the output is a visible light image with annotated bounding boxes and mask images; S5. Train the basic model on the data set {(visible light image, auxiliary image); (mask image, bounding box)} to update the attention feature extraction module, convolution module and Mask-RCNN model until the basic model converges; the converged basic model is the remote sensing landslide target detection model.

2. The training method for the remote sensing landslide target detection model according to claim 1, characterized in that: The attribute characteristics include one or more of elevation, slope, aspect, plan curvature, profile curvature, vegetation coverage, annual rainfall, water flow intensity index and terrain wetness index.

3. The training method for the remote sensing landslide target detection model according to claim 1, characterized in that: The basic model is trained on the dataset, and the loss function used in the training process is the sum of classification loss, bounding box regression loss and mask segmentation loss.

4. The training method for the remote sensing landslide target detection model according to claim 3, characterized in that: Classification loss Loss cls The calculation formula is: Among them, y i is the true category label, y i =1 means landslide is detected, y i =0 means no landslide was detected; p i represents the landslide probability predicted by the basic model; N is the number of categories, including those with landslide and those without landslide.

5. The training method for the remote sensing landslide target detection model according to claim 3, characterized in that: Bounding box regression loss Loss box The calculation formula is: Where M is the number of landslide sample anchor frames; t mn Represents the basic model prediction value of the regression target n of the mth landslide sample anchor frame. The regression target includes the center coordinates (x, y) of the anchor frame, the width w' of the anchor frame, and the height h', t' of the anchor frame mn represents the true value of the regression target n of the mth landslide sample anchor box; (t mn -t' mn ) indicates t mn and t' mn ; SmoothL1 is the value selection function; x' is a proxy parameter.

6. The training method for the remote sensing landslide target detection model according to claim 3, characterized in that: Mask segmentation loss mask The calculation formula is: Where H is the height of the visible light image, W is the width of the visible light image; y h,w represents the value of the pixel coordinate (h, w) on the real mask image corresponding to the visible light image; p h,w It represents the probability that the pixel coordinate (h, w) value on the mask image predicted by the basic model is 1; ln represents the logarithmic function.

7. The training method for the remote sensing landslide target detection model according to claim 1, characterized in that: The embedding module uses query dictionary embedding; the position encoding module uses rotation encoding, and the attention feature extraction module uses CBAM convolutional attention module.

8. A remote sensing landslide target detection method using the remote sensing landslide target detection model training method according to any one of claims 1 to 7, characterized in that: include: St1. Acquire a remote sensing landslide target detection model using the remote sensing landslide target detection model training method as described in any one of claims 1 to 7; acquire a visible light image and an auxiliary image of the detection area; St2. Input the visible light image and the auxiliary image into the remote sensing landslide target detection model, and the remote sensing landslide target detection model outputs the visible light image with the marked bounding box and mask image.

9. A remote sensing landslide target detection system, characterized in that: The invention comprises an unmanned aerial vehicle, a memory and a processor, wherein the unmanned aerial vehicle is used to collect visible light images and auxiliary images; a computer program is stored in the memory, and the processor is connected to the memory and the unmanned aerial vehicle, and the processor is used to execute the computer program to implement the remote sensing landslide target detection method as described in any one of claims 1 to 8.

10. A readable medium, characterized in that: A computer program is stored, and when the computer program is executed, it is used to implement the remote sensing landslide target detection method according to any one of claims 1 to 8.

Images