Landslide mass identification method, device, equipment, medium and program product

Abstract

The invention provides a landslide mass recognition method and device, equipment, a storage medium and a program product, and can be applied to the field of landslide mass recognition. The method comprises the following steps: based on a landslide mass identification model, respectively carrying out feature extraction on images from a plurality of modalities to obtain extraction features corresponding to the images; converting the extracted features into corresponding frequency domains and carrying out frequency domain compression to obtain frequency domain features corresponding to the image; converting the frequency domain features into weight factors, and obtaining weighted features corresponding to the images according to the weight factors and extraction features corresponding to the same image; splicing the weighted features of the images to obtain fusion features; processing the fused features through a multilayer dense network, and carrying out multiple interpolations on the processed features by using a bilinear interpolation method; and obtaining the category of each pixel in the features after multiple interpolations by using a maximum parameter value decision method, and determining the landslide mass.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of landslide body identification, in particular to a landslide body identification method, device, equipment, medium and program product. BACKGROUND

[0002] Landslide is one of the natural disasters with extremely strong harmfulness, and the identification of landslide body can reduce the loss caused by natural disasters. At present, the identification method of landslide body mainly fuses multiple modal features linearly, and then screens the redundant information in each modal feature by using a gating mechanism to obtain the landslide body region. This method has the following defects: (1) linear fusion of each feature cannot fully integrate the interactive features of multiple modal features, and different modal features are very important in the identification process of landslide body, so this defect seriously reduces the accuracy of landslide body identification. (2) the gating mechanism cannot extract the relationship between different modal features, and in the model training process, it is easily affected by the difference between different modal features, resulting in poor model training effect, and further affecting the identification accuracy of landslide body. In summary, the existing landslide body identification method is prone to attention shift due to too large statistical distribution difference between modalities when facing high-dimensional heterogeneous data, resulting in excessive attention to redundant information of a certain modality and ignoring key information of other modalities, which ultimately leads to failure to correctly identify the landslide body. Moreover, the matching effect of the existing landslide body identification method on cross-modal data needs to be improved, and the noise interference is large. SUMMARY

[0003] In view of the above problems, the present application provides a landslide body identification method, device, equipment, medium and program product which can improve the identification accuracy of landslide body and reduce the computational complexity.

[0004] According to a first aspect of the present application, a landslide body identification method is provided, comprising: based on a landslide body identification model, respectively extracting features from images derived from multiple modalities to obtain respective extraction features corresponding to the images, wherein the extraction features are obtained through multiple feature extraction network layers; transforming the extraction features to a corresponding frequency domain and performing frequency domain compression to obtain frequency domain features corresponding to the images; converting the frequency domain features into weight factors, and according to the weight factors and the extraction features corresponding to the same image, obtaining weighted features corresponding to each image; splicing the weighted features corresponding to each image to obtain fusion features; processing the fusion features through a multi-layer dense network, and performing multiple interpolations on the processed features using a bilinear interpolation method; using a maximum parameter value decision method to obtain the categories of each pixel in the multiple interpolated features, and determining the landslide body.

[0005] According to an embodiment of the present application, feature extraction is performed on images from multiple modalities respectively to obtain respective extracted features of the images, including: for any multi-modal image, performing convolution operation on the multi-modal image to obtain a first feature vector; passing the first feature vector through a plurality of feature extraction network layers respectively to obtain a plurality of second feature vectors, wherein the feature extraction network layers include at least one first normal convolution layer, at least one dilated convolution layer, and an average pooling layer; and splicing the plurality of second feature vectors to obtain the extracted feature corresponding to the multi-modal image.

[0006] According to an embodiment of the present application, the extracted features are transformed into a corresponding frequency domain and compressed in the frequency domain to obtain respective frequency domain features of the images, including: dividing the extracted features into a plurality of parts along the channel dimension, and mapping the plurality of parts to the frequency domain according to a discrete cosine transform basis function, wherein each part corresponds to a frequency index, and the discrete cosine transform basis function is constructed according to the frequency index; performing feature compression on the feature vectors in the frequency domain, filtering the feature vectors after feature compression to obtain the frequency domain features.

[0007] According to an embodiment of the present application, the frequency domain features are converted into weight factors, and the weight factors and the extracted features corresponding to the same image are used to obtain weighted features corresponding to each image, including: inputting the frequency domain features into a second normal convolution layer to obtain the weight factors; multiplying the weight factors and the extracted features corresponding to the same image to obtain the weighted features corresponding to the image; and performing the weighted feature acquisition operation on the frequency domain features corresponding to any image to obtain the weighted features corresponding to each image.

[0008] According to an embodiment of the present application, the processed features are interpolated multiple times using a bilinear interpolation method, including: performing first interpolation on the processed features using the bilinear interpolation method to obtain first interpolated features; fusing the first interpolated features and the fused features, and performing second interpolation on the fused features.

[0009] According to an embodiment of the present application, the class of each pixel in the features after multiple interpolations is obtained using a maximum parameter value decision method to determine a landslide body, including: inputting the features after multiple interpolations into a class convolution layer, performing normalization processing on the output of the class convolution layer to obtain the probability of the class to which each pixel belongs, wherein the class includes a landslide body, a crack zone, a loose accumulation area, and a background area; determining the class of each pixel based on the maximum parameter value decision method and the probability; and integrating the pixels whose class belongs to the landslide body to obtain the region where the landslide body is located.

[0010] The second aspect of the present application provides a landslide body identification device, comprising: a feature extraction module, configured to perform feature extraction on images from multiple modalities based on a landslide body identification model, respectively, to obtain respective extracted features corresponding to the images, wherein the extracted features are obtained through multiple feature extraction network layers; a frequency domain feature acquisition module, configured to transform the extracted features into corresponding frequency domains and perform frequency domain compression to obtain respective frequency domain features corresponding to the images; a weighted feature acquisition module, configured to convert the frequency domain features into weight factors, and obtain weighted features corresponding to each image according to the weight factors and the extracted features corresponding to the same image; a feature splicing module, configured to splice the weighted features corresponding to each image to obtain fused features; and a spliced feature processing module, configured to process the fused features through a multi-layer dense network and perform multiple interpolations on the processed features using a bilinear interpolation method. A landslide body identification module is configured to determine the landslide body by obtaining the categories of each pixel in the interpolated features using a maximum parameter value decision method.

[0011] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the steps of the above method.

[0013] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the above method. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a landslide body identification method according to an embodiment of the present application is schematically shown; Figure 2 A training method schematic diagram of a landslide body identification model according to an embodiment of the present application is schematically shown; Figure 3 A feature extraction method schematic diagram according to an embodiment of the present application is schematically shown; Figure 4 A weighted feature acquisition method schematic diagram according to an embodiment of the present application is schematically shown; Figure 5 A pixel category determination method schematic diagram according to an embodiment of the present application is schematically shown; Figure 6 A flowchart of an interpolation method according to an embodiment of the present application is schematically shown; Figure 7 A flowchart of a method of obtaining a region where a landslide body is located according to an embodiment of the present application is schematically shown; Figure 8 A block diagram of a structure of a landslide body identification device according to an embodiment of the present application is schematically shown; Figure 9 A block diagram of an electronic device suitable for implementing a landslide body identification method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0015] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely illustrative and is not intended to limit the scope of the present application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0016] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall not be taken to exclude

[0017] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and not be interpreted in an idealized or overly formal way.

[0018] In the case where expressions similar to "at least one of A, B, and C, and the like" are used, it is generally to be interpreted as including one or more of the items enumerated in the expression (e.g., "a system having at least one of A, B, and C" should include, but not be limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0019] Embodiments of the present application provide a landslide body identification method, which obtains extracted features of images from each modality through a plurality of feature extraction network layers, expands the receptive field from local to global, and improves the recognition ability of the model; maps the extracted features to the frequency domain, greatly reduces the attention shift problem from multiple different data sources, compresses spatial information into a global scalar using frequency domain compression, and greatly reduces the computational complexity of the model; based on the fusion features, interpolation is performed using a bilinear interpolation method, which improves the perception ability of the model to context semantic information, effectively alleviates the boundary drift and class confusion problems, and greatly improves the recognition accuracy.

[0020] Figure 1 A flowchart of a landslide body identification method according to an embodiment of the present application is schematically shown.

[0021] As shown in Figure 1 the landslide body identification method 100 of this embodiment includes operations S110-S160.

[0022] In operation S110, based on a landslide body identification model, features are extracted from images from multiple modalities respectively, to obtain respective extracted features corresponding to the images, wherein the extracted features are obtained through a plurality of feature extraction network layers.

[0023] In operation S120, the extracted features are transformed to the corresponding frequency domain and compressed in the frequency domain to obtain respective frequency domain features corresponding to the images.

[0024] In operation S130, the frequency domain features are converted into weight factors, and based on the weight factors and the extracted features corresponding to the same image, weighted features corresponding to each image are obtained.

[0025] In operation S140, the weighted features corresponding to each image are spliced to obtain fusion features.

[0026] In operation S150, the fusion features are processed through a multi-layer dense network, and the processed features are interpolated multiple times using a bilinear interpolation method.

[0027] In operation S160, the class of each pixel in the features after multiple interpolations is obtained using a maximum parameter value decision method, and the landslide body is determined.

[0028] In some embodiments, in operation S120, the image can be an image of a certain region derived from a synthetic aperture radar (SAR) acquisition, referred to as a SAR image, or an optical image derived from the region, referred to as an optical image. After inputting the images derived from multiple modalities (such as SAR images and optical images) into the landslide body recognition model, pixel registration needs to be performed on the SAR images and the optical images to ensure semantic consistency. Specifically, the SAR images and the optical images of the same time and the same region are unified into the same geographic coordinate system, coarse alignment is performed on the SAR images and the optical images through an imaging geometric model or based on a radial distortion correction parameter, an image pair is formed, same-name feature points that are stably present in the SAR images and the optical images are obtained from the image pair, based on a mismatch elimination mechanism, the mismatched feature points in the same-name feature points are removed, high-precision same-name feature points are obtained, a pixel-level spatial transformation relationship between the SAR images and the optical images is calculated according to the high-precision same-name feature points, the SAR images are resampled according to the transformation relationship, so that the row and column distributions and the pixel sizes of the SAR images are consistent with those of the optical images, and finally, the registration accuracy is calculated through mutual information or a peak signal-to-noise ratio or visual inspection, so as to ensure that the SAR images and the optical images achieve pixel-level consistency in space.

[0029] Figure 2 A schematic diagram of a training method of a landslide body recognition model according to an embodiment of the present application is schematically shown.

[0030] As Figure 2As shown, the training process of the landslide body identification model includes three parts, which are landslide body dataset construction part, multi-scale frequency domain attention fusion part and decoding output part. The landslide body dataset construction part includes pixel registration, data enhancement and data labeling operations. The multi-scale frequency domain attention fusion part includes multi-scale dilated feature extraction and frequency domain attention weighting operations. The decoding output part includes dense connection, bilinear interpolation and pixel classification operations. Specifically, in the training process of the landslide body identification model, first, the historical SAR images and the historical optical images in the same time and the same region are pixel-registered. Then, the pixel-registered historical SAR images and the historical optical images are synchronously geometrically transformed, which includes random horizontal flipping, vertical flipping, implementing 90° or 180° or 270° integer multiple rotation at the same angle and synchronous cropping. In this way, the training image pairs that are spatially aligned and have the same size are obtained. The training image pairs are pixel-level semantic labeled, that is, according to the semantic information indicated by each pixel in the training image pairs, the corresponding label is labeled, and the label includes landslide body, fracture zone, loose accumulation area and background area. All labeled labels are stored in an 8-byte single-channel mask. For example, the label corresponding to the background area is 0, and the labels of the other three types are 1, 2 and 3 respectively. Then, the multi-scale dilated feature extraction and frequency domain attention weighting of the training image pairs are performed through the multi-scale frequency domain attention fusion part. That is, the extraction features of each image in the training image pairs are obtained through multiple feature extraction network layers. For example, two first normal convolution layers are used to capture the details and textures of each image, two dilated convolution layers are used to expand the receptive field, and an average pooling layer is used to supplement the scene-level semantic information. The output of each first normal convolution layer, the output of each dilated convolution layer and the output of the average pooling layer are spliced to obtain the extraction features corresponding to each image. The dilation rates of the two dilated convolution layers are incremental and non-multiple empty convolution layers. The final extraction features contain both fracture-level details and hundred-meter-level contextual hierarchical features. This process is multi-scale dilated feature extraction. Next, frequency domain attention weighting is performed. First, the above extraction features are transformed into the frequency domain and compressed in the frequency domain to obtain frequency domain features. Then, the frequency domain features are converted into weight factors. Based on the weight factors and the extraction features of the same image in the training image pairs, the weighted image features are obtained. In this way, the complementary enhancement and redundancy suppression of cross-modal information are realized.Finally, the weighted image features are densely connected, bilinear interpolated and pixel classified by the decoding output part. Specifically, the weighted image features are input into a multi-layer dense network for processing to fully reuse high-level semantics, then multiple bilinear interpolations are used to gradually restore high-resolution spatial details, and then a class convolution layer is used to obtain the probability of each pixel belonging to a class in the training image pair. Based on the maximum parameter value decision method and the obtained probability, the landslide body, fracture zone, loose accumulation area and background area are finally predicted. By comparing the above prediction results with the labeled results, the gap between them is calculated, and based on the gap, a comprehensive loss function is calculated. The model parameters are adjusted based on the comprehensive loss function, and based on the adjusted model parameters, the above training process is repeated until the comprehensive loss function converges. In this way, the model can balance class balance and boundary accuracy, and the model can still maintain high robustness and high-fine-grained landslide body segmentation capability in complex water area scenes. The expression of the comprehensive loss function is as follows: ; , and represent the cross-entropy loss function, the focal loss function and the IoU loss function respectively, , and are the hyperparameters corresponding to the three loss functions respectively. The cross-entropy loss function is used to measure the difference between the model prediction probability distribution and the true label distribution, and its expression is as follows: ; represents the true label value of the image belonging to the th class, represents the probability value of the image belonging to the th class predicted by the model, indicates the number of classes, for example, in the case of classifying landslide body, fracture zone, loose accumulation area and background area, the value of is 3; the focal loss function improves the classification accuracy of difficult-to-classify samples by introducing a hyperparameter , that is, it can enhance the recognition accuracy of the model for sporadically distributed landslide bodies, and its expression is as follows: ; wherein is another hyperparameter, in some embodiments, can be taken as 1,The intersection over union loss function is used to optimize the intersection over union of the predicted value and the true label value, so as to improve the accuracy of boundary alignment and object shape recognition. For a landslide with a fuzzy boundary, the intersection over union loss function can ensure that the model learns geometric consistency and avoid “jagged” segmentation. It pays more attention to the consistency of the global structure. The expression of the intersection over union loss function is as follows: ; wherein, is a true labeled landslide region, is a landslide region predicted by the model.

[0031] Figure 3 An extraction feature method according to an embodiment of the present application is schematically shown.

[0032] Because SAR is based on backscattering, sensitive to structure, dielectric constant, and optical dependent on solar reflection, providing texture, color information, it can be seen that the heterogeneity difference between the optical image and the SAR image is large, so they are not suitable for direct pixel-level fusion, therefore, the features of each image can be extracted first, and then fused by a certain strategy. In some embodiments, the features of the images from multiple modalities are extracted respectively to obtain the extracted features corresponding to the images respectively, including: for any image from multiple modalities, performing convolution operation on the multiple modalities image to obtain a first feature vector; the first feature vector is respectively input into a plurality of feature extraction network layers to obtain a plurality of second feature vectors, wherein the feature extraction network layer includes at least one first ordinary convolution layer, at least one dilated convolution layer and an average pooling layer; and the plurality of second feature vectors are spliced to obtain the extracted features corresponding to the multiple modalities image.

[0033] As Figure 3 shown, for an image from a certain modality (such as a SAR image or an optical image), the image is first convolved by a convolution layer to obtain a first feature vector, and the size of the first feature vector is , wherein, is the number of feature units of the first feature vector in the vertical direction, is the number of feature units of the first feature vector in the horizontal direction, is the number of feature units of the image in the channel dimension, which can moderately enhance the feature diversity, then the first feature vector is input into 5 parallel feature extraction network layers to obtain 5 second feature vectors, each feature extraction network layer outputs a second feature vector, and the 5 parallel feature extraction network layers are two first ordinary convolution layers, two dilated convolution layers and an average pooling layer, finally the 5 second feature vectors obtained are spliced (i.e. Figure 3 ​The operation yields the extracted features of the image, and the scale of these extracted features is [scale value missing]. , , and These five independent feature extraction network layers, each with multiple receptive fields, represent the number of feature units extracted in the vertical direction, the number of feature units extracted in the horizontal direction, and the number of feature units extracted in the channel dimension. These layers can improve the model's ability to perceive objects of different sizes in the image. Two first ordinary convolutional layers are used to extract more details, with 3 and 5 convolutional kernels, respectively. Two dilated convolutional layers can increase the receptive field and extract more contextual content. The dilated convolutional layers expand the receptive field by inserting holes (i.e., gaps) into the standard convolutional kernels. Through these feature extraction network layers, convolutional operations at different scales can be achieved, which is beneficial for capturing multi-scale features.

[0034] Furthermore, when the expansion rate is When the equivalent convolution kernel size is: ; in To expand the kernel size, This represents the original convolution kernel size. Different dilation rates directly affect the segmentation results. The grid effect of dilated convolution refers to the possibility that pixels in certain regions of the first feature vector may be ignored or not effectively utilized when using the same or multiple dilation rates consecutively, leading to a disruption of information continuity. To avoid this problem, this application adopts a hybrid dilated convolution criterion, setting the dilation rate as a scaling structure. To avoid gaps between pixels that could lead to information loss, the dilation rate setting requirements are as follows: ; in, Indicates the first The "effective coverage constraint" of dilated convolution refers to the degree of continuous pixel coverage after passing through the dilated convolution layer. Indicates the first "Effective Coverage Constraint" of Layer Dilated Convolution Indicates the first The dilation rate of the layer dilated convolution is given by the following constraint: and ,in, Indicates the first "Effective Coverage Constraint" of Layer Dilated Convolution Indicates the first The dilation rate of the layer dilated convolution, the first This is the last layer of the dilated convolutional layer. To preset parameters, in this application, based on the above expansion rate setting requirements, the expansion rate sequence of the first expansion convolution layer is [1, 2, 3], and the expansion rate sequence of the second expansion convolution layer is [1, 2, 5]. Based on the receptive field calculation formula, it can be obtained that the receptive field after the first layer of expansion convolution layer is expanded to , the receptive field after the second layer of expansion convolution layer is expanded to , and the second feature vector is obtained after the last average pooling layer, which realizes further extraction of global information. By splicing 5 second feature vectors, it can be ensured that the features extracted by each feature extraction network layer can be retained, rather than compressed features or fused features.

[0035] Figure 4 The weighted feature acquisition method according to the embodiment of the application is schematically shown.

[0036] The acquisition method of the weighted feature is mainly through Discrete Cosine Transform (DCT) to convert the extracted feature from the spatial domain to the frequency domain for attention calculation, so as to effectively capture the global frequency domain feature and suppress noise interference. In some embodiments, the extracted feature is transformed into the corresponding frequency domain and compressed in the frequency domain to obtain the frequency domain feature corresponding to each image, including: dividing the extracted feature into multiple parts along the channel dimension, mapping the multiple parts to the frequency domain according to the Discrete Cosine Transform basis function, wherein each part corresponds to a frequency index, and the Discrete Cosine Transform basis function is constructed according to the frequency index; performing feature compression on the feature vector in the frequency domain, filtering the feature vector after feature compression to obtain the frequency domain feature. In some embodiments, the frequency domain feature is converted into a weight factor, and the weighted feature corresponding to each image is obtained according to the weight factor and the extracted feature corresponding to the same image, including: inputting the frequency domain feature into a second ordinary convolution layer to obtain the weight factor; multiplying the weight factor and the extracted feature corresponding to the same image to obtain the weighted feature corresponding to the image; and performing the weighted feature acquisition operation on the frequency domain feature corresponding to any image to obtain the weighted feature corresponding to each image.

[0037] As shown in Figure 4 , taking an optical image as an example, the extracted feature corresponding to the optical image is divided into parts along the channel dimension, and each part has a size of , The number of feature units in the channel dimension of each part is , ,..., These feature vectors are filtered according to a preset filtering rule to obtain frequency domain features, and then convolution is performed on the frequency domain features. conv The weighting factor is obtained by multiplying it with the extracted features corresponding to the optical image to obtain the weighted feature corresponding to the optical image. The scale of the weighted feature is the same as the scale of its corresponding extracted feature. Further, the preset filtering rule can be: for the optical image, a mid-frequency index is used for bandpass filtering to preserve the edge and texture features of the landslide body; for the SAR image, a low-frequency index is used for low-pass filtering to preserve the overall structure and smooth areas of the landslide body. Each part corresponds to a DCT frequency index. , Indicates the first The frequency index value of each part along the horizontal direction, Indicates the first Frequency index values ​​of each part along the vertical direction. Further, using... This indicates feature extraction, along the channel dimension. Divided into Each part needs to meet the channel dimension. Can be Divisible, and .

[0038] Furthermore, the expressions for the discrete cosine transform basis functions are as follows: ; in, , These represent the spatial position index values ​​in the horizontal direction and the vertical direction, respectively. , The formula for feature compression is as follows: ; in, and These represent the th element after DCT transformation. The characteristics of each part in the frequency domain and the first The spatial location of each part The feature vector at a given location is compressed into a scalar vector through feature compression, which greatly reduces computational complexity while preserving frequency domain information. By concatenating the frequency domain features of each part and then filtering, the frequency domain features containing global information can be obtained. .

[0039] Will An input one-dimensional second general convolution layer is used for convolution to obtain a weight factor. Compared with a full connection layer, the one-dimensional convolution parameter amount is significantly reduced, and has stronger spatial flexibility, which is suitable for mobile deployment. After obtaining the weight factor, a normalization function is used for normalization processing to maintain semantic consistency. The expression of the normalization function is as follows: . wherein, is a normalized exponential function, is a weight factor, is multiplied by to obtain a weighted feature , , The response of an important channel is enhanced in the middle, so that the weighted feature of the optical image is obtained. The same method can also be used to obtain the weighted feature of the SAR image. The and are spliced to obtain a fusion feature .

[0040] By mapping the spatial domain feature to the frequency domain through discrete cosine transform, an adaptive band-pass filter is designed in the frequency dimension to selectively retain the key frequency (low frequency) of the terrain structure and suppress the speckle noise (high frequency). In the frequency domain, the overall contour and main trend of the landslide body and other large-scale terrain structure information are mainly concentrated in the low-frequency component, while the high-frequency information such as sensor noise, water surface reflection and vegetation texture is represented as discrete spectral peaks. By adopting different frequency index setting strategies, the optical and SAR modal branches use different frequency band-pass filters: the optical branch uses a medium frequency band-pass filter to retain rock texture and suppress water surface reflection; the SAR branch uses a low frequency band-pass filter to suppress speckle noise and highlight the overall structure.

[0041] Figure 5 A schematic diagram of a pixel category determination method according to an embodiment of the application is shown schematically.

[0042] As shown in Figure 5 , the fusion feature is first input into a multi-layer dense network. The number of layers of the multi-layer dense network used in the application is 3 layers. After passing through the multi-layer dense network, the semantic feature can be further improved. The output of the multi-layer dense network is input into a first bilinear interpolation layer for bilinear interpolation. The scale of the feature after the first interpolation is . The feature after the first interpolation is connected to the fusion feature through a skip connection, that is, the features are fused. The fused feature is input into a second bilinear interpolation layer for bilinear interpolation to obtain the scale of the feature after the second interpolation ., through the multiple bilinear interpolation operations, the feature propagation and the feature reuse can be enhanced, the weight parameters of the model can be reduced, the information sharing can be improved, the model has good performance in accuracy and generalization ability, through the jump connection, the utilization rate of the multi-modal features can be improved, and a more comprehensive segmentation result can be obtained; the feature after the second interpolation is input into the category convolution layer, the scale of the feature output by the category convolution layer is , the category convolution layer can obtain the probability of the category to which each pixel belongs, and an argument of the maximum (argmax) is used to obtain a semantic segmentation map, and the semantic segmentation map indicates the category to which each image belongs.

[0043] Figure 6 An interpolation method flowchart according to an embodiment of the application is schematically shown.

[0044] As shown in Figure 6 , the interpolation method includes operations S610-S620.

[0045] In operation S610, the bilinear interpolation method is used to perform first interpolation on the processed feature to obtain first interpolation feature.

[0046] In operation S620, the first interpolation feature is fused with the fusion feature, and the fused feature is subjected to second interpolation.

[0047] In some embodiments, before the above interpolation operations are performed, the fusion feature is input into a multi-layer dense network for processing. If the number of layers of the multi-layer dense network is , the output of the layer dense network is a splicing of the output results of all previous dense networks, and the expression is as follows: ; Wherein, represents a splicing operation, , and respectively represent the output feature of the first layer dense network, the output feature of the second layer dense network and the output feature of the layer dense network, each layer of the dense network includes a convolution layer, a normalization layer and an activation function layer, and each layer of the dense network can obtain the output of the layer of the dense network through the convolution layer, the normalization layer and the activation function layer.

[0048] Figure 7 A landslide region acquisition method flowchart according to an embodiment of the application is schematically shown.

[0049] As shown in Figure 7As shown, the landslide body region acquisition method includes operation S710 to operation S730.

[0050] In operation S710, the multiple times interpolated features are input into a category convolution layer, and the output of the category convolution layer is normalized to obtain the probability of the category to which each pixel belongs, wherein the category includes a landslide body, a fissure zone, a loose accumulation area, and a background area.

[0051] In operation S720, the category of each pixel is determined based on the maximum parameter value decision method and the probability.

[0052] In operation S730, pixels belonging to the category of the landslide body are integrated to obtain the region where the landslide body is located.

[0053] In some embodiments, in operation S710, the multiple times interpolated features are convolved by the following formula: ; wherein, represents the output of the category convolution layer, represents a one-dimensional category convolution operation, represents the multiple times interpolated features. The probability of the category to which each pixel belongs is obtained by the following formula: ; wherein, represents the probability that the pixel with the pixel position is classified into the category , represents the key information of the category extracted from the pixel with the pixel position in , represents the key information of the category extracted from the pixel with the pixel position in . The expression used to determine the pixel category based on the maximum parameter value decision method is as follows: ; wherein, represents the category to which the pixel with the position belongs, represents the maximum parameter value decision operation, and the category to which the pixel with the position belongs can be a landslide body or a fissure zone or a loose accumulation area or a background area.

[0054] Based on the above landslide body identification method, the application further provides a landslide body identification device. The device will be described in detail below. Figure 8

[0055] ​Figure 8 A structural block diagram of a landslide body identification apparatus according to an embodiment of the present application is shown schematically.

[0056] As shown in Figure 8 The landslide body identification apparatus 800 of this embodiment includes a feature extraction module 810, a frequency domain feature acquisition module 820, a weighted feature acquisition module 830, a feature splicing module 840, a spliced feature processing module 850, and a landslide body identification module 860.

[0057] The feature extraction module 810 is configured to perform feature extraction on images from multiple modalities based on a landslide body identification model, respectively, to obtain respective extraction features corresponding to the images, wherein the extraction features are obtained through multiple feature extraction network layers. In an embodiment, the feature extraction module 810 can be configured to perform the operation S110 described above, and thus no further description is given here.

[0058] The frequency domain feature acquisition module 820 is configured to transform the extraction features to corresponding frequency domains and perform frequency domain compression to obtain respective frequency domain features corresponding to the images. In an embodiment, the frequency domain feature acquisition module 820 can be configured to perform the operation S120 described above, and thus no further description is given here.

[0059] The weighted feature acquisition module 830 is configured to convert the frequency domain features into weight factors, and obtain respective weighted features corresponding to the images according to the weight factors and the extraction features corresponding to the same images. In an embodiment, the weighted feature acquisition module 830 can be configured to perform the operation S130 described above, and thus no further description is given here.

[0060] The feature splicing module 840 is configured to splice the respective weighted features corresponding to the images to obtain fused features. In an embodiment, the feature splicing module 840 can be configured to perform the operation S140 described above, and thus no further description is given here.

[0061] The spliced feature processing module 850 is configured to process the fused features through a multi-layer dense network and perform multiple interpolations on the processed features using a bilinear interpolation method. In an embodiment, the spliced feature processing module 850 can be configured to perform the operation S150 described above, and thus no further description is given here.

[0062] The landslide body identification module 860 is configured to obtain the categories of each pixel in the features after the multiple interpolations using a maximum parameter value decision method, and determine a landslide body. In an embodiment, the landslide body identification module 860 can be configured to perform the operation S160 described above, and thus no further description is given here.

[0063] According to the embodiments of the present application, the landslide body recognition device 800 can obtain the extracted features from the images of various modalities, expand the receptive field from local to global, and improve the recognition ability of the model; the landslide body recognition device 800 can map the extracted features to the frequency domain, greatly reduce the attention shift problem from multiple different data sources, and compress the spatial information into a global scalar, greatly reducing the computational complexity of the model; the interpolation method based on the landslide body recognition device 800 can improve the perception ability of the model to the context semantic information, effectively alleviate the boundary drift and class confusion problems, and greatly improve the recognition accuracy.

[0064] In some embodiments, the feature extraction module 810 is specifically configured to: for any source of multi-modal images, performing convolution operation on the multi-modal images to obtain a first feature vector; passing the first feature vector through a plurality of feature extraction network layers respectively to obtain a plurality of second feature vectors, wherein the feature extraction network layer includes at least one first normal convolution layer, at least one dilated convolution layer, and an average pooling layer; and splicing the plurality of second feature vectors to obtain the extracted features corresponding to the multi-modal images.

[0065] In some embodiments, the frequency domain feature acquisition module 820 is specifically configured to: divide the extracted features into a plurality of parts along the channel dimension, and map the plurality of parts to the frequency domain according to a discrete cosine transform basis function, wherein each part corresponds to a frequency index, and the discrete cosine transform basis function is constructed according to the frequency index; performing feature compression on the feature vectors in the frequency domain, filtering the feature vectors after feature compression to obtain the frequency domain features.

[0066] In some embodiments, the weighted feature acquisition module 830 is specifically configured to: input the frequency domain features into a second normal convolution layer to obtain a weight factor; multiply the weight factor with the extracted features corresponding to the same image to obtain the weighted features corresponding to the image; and perform the weighted feature acquisition operation on the frequency domain features corresponding to any image to obtain the weighted features corresponding to each image.

[0067] In some embodiments, the splicing feature processing module 850 is specifically configured to: perform first interpolation on the processed features using the bilinear interpolation method to obtain first interpolated features; fuse the first interpolated features with the fused features, and perform second interpolation on the fused features.

[0068] In some embodiments, the landslide body recognition module 860 is specifically configured to: input the features after multiple interpolations into a category convolution layer, perform normalization processing on the output of the category convolution layer to obtain the probability of the category to which each pixel belongs, wherein the category includes landslide body, crack zone, loose accumulation area, and background area; determine the category of each pixel based on the maximum parameter value decision method and the probability; and integrate the pixels whose category is landslide body to obtain the area where the landslide body is located.

[0069] According to an embodiment of the present application, any multiple of the feature extraction module 810, the frequency domain feature acquisition module 820, the weighted feature acquisition module 830, the feature splicing module 840, the spliced feature processing module 850 and the landslide body identification module 860 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules and implemented in one module. According to an embodiment of the present application, at least one of the feature extraction module 810, the frequency domain feature acquisition module 820, the weighted feature acquisition module 830, the feature splicing module 840, the spliced feature processing module 850 and the landslide body identification module 860 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of the three implementation ways of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the feature extraction module 810, the frequency domain feature acquisition module 820, the weighted feature acquisition module 830, the feature splicing module 840, the spliced feature processing module 850 and the landslide body identification module 860 can be at least partially implemented as a computer program module which can perform corresponding functions when it is run.

[0070] Figure 9 The block diagram of an electronic device suitable for implementing the landslide body identification method according to an embodiment of the present application is schematically shown.

[0071] As shown in Figure 9 The electronic device 900 according to an embodiment of the present application includes a processor 901 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0072] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via the bus 904. The processor 901 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0073] According to the embodiments of the present application, the electronic device 900 can further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 can further include one or more of the following components connected to the input / output (I / O) interface 905: an input part 906 including a keyboard, a mouse, and the like; an output part 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 908 including a hard disk, and the like; and a communication part 909 including a network interface card such as a LAN card, a modem, and the like. The communication part 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 910 as necessary, so that a computer program read therefrom is installed in the storage part 908 as necessary.

[0074] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0075] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include one or more of the above-described ROM 902 and / or RAM 903 and / or a memory other than the ROM 902 and the RAM 903.

[0076] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the landslide body identification method provided by the embodiments of the present application.

[0077] The above-described functions defined in the system / device / apparatus of the embodiments of the present application are performed when the computer program is executed by the processor 901. According to an embodiment of the present application, the above-described system, device, module, unit, etc. can be implemented by computer program modules.

[0078] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 909, and / or be installed from the detachable medium 911. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

[0079] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or be installed from the detachable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the above-described system, device, apparatus, module, unit, etc. can be implemented by computer program modules.

[0080] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C", or the like. Program code can execute entirely on a user's computing device, partly on the user's device, as a stand-alone software package, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0081] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0082] Those skilled in the art will appreciate that features recited in the various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations have not been explicitly recited in the present application. In particular, the features recited in the various embodiments of the present application can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present application. All such combinations are within the scope of the present application.

Claims

1. A method for identifying landslide bodies, characterized in that, The method includes: Based on the landslide body recognition model, feature extraction is performed on images from multiple modalities to obtain the extracted features corresponding to each image. The extracted features are obtained through multiple feature extraction network layers. The extracted features are transformed to the corresponding frequency domain and compressed in the frequency domain to obtain the frequency domain features corresponding to each image. The frequency domain features are converted into weighting factors, and the weighted features corresponding to each image are obtained based on the weighting factors and the extracted features corresponding to the same image. By concatenating the weighted features corresponding to each image, a fused feature is obtained; The fused features are processed by a multi-layer dense network, and the processed features are interpolated multiple times using bilinear interpolation. The category of each pixel in the features after multiple interpolations is obtained by using the maximum parameter value decision method, and the landslide body is determined.

2. The method according to claim 1, characterized in that, The step of extracting features from images from multiple modalities to obtain the extracted features corresponding to each image includes: For a multimodal image from any source, perform a convolution operation on the multimodal image to obtain a first feature vector; The first feature vector is passed through the plurality of feature extraction network layers to obtain a plurality of second feature vectors, wherein the feature extraction network layer includes at least one first ordinary convolutional layer, at least one dilated convolutional layer and an average pooling layer; The extracted features corresponding to the multimodal image are obtained by concatenating the multiple second feature vectors.

3. The method according to claim 1, characterized in that, The step of transforming the extracted features to the corresponding frequency domain and performing frequency domain compression to obtain the frequency domain features corresponding to each image includes: The extracted features are divided into multiple parts along the channel dimension, and the multiple parts are mapped to the frequency domain according to the discrete cosine transform basis function, wherein each part corresponds to a frequency index, and the discrete cosine transform basis function is constructed according to the frequency index; The feature vector in the frequency domain is compressed, and the compressed feature vector is filtered to obtain the frequency domain features.

4. The method according to claim 1, characterized in that, The step of converting the frequency domain features into weighting factors, and obtaining weighted features for each image based on the weighting factors and the extracted features corresponding to the same image, includes: The frequency domain features are input into the second ordinary convolutional layer to obtain the weighting factors; Multiply the weight factor by the extracted feature corresponding to the same image to obtain the weighted feature corresponding to that image; For any of the frequency domain features corresponding to the image, the weighted feature acquisition operation is performed to obtain the weighted features corresponding to each image.

5. The method according to claim 1, characterized in that, The process of interpolating the processed features multiple times using bilinear interpolation includes: The bilinear interpolation method is used to perform a first interpolation on the processed features to obtain the first interpolated features. The first interpolation feature and the fused feature are merged, and the fused feature is then interpolated a second time.

6. The method according to claim 1, characterized in that, The step of using the maximum parameter value decision method to obtain the category of each pixel in the features after multiple interpolations and determining the landslide body includes: The features obtained after multiple interpolations are input into the category convolutional layer, and the output of the category convolutional layer is normalized to obtain the probability of the category to which each pixel belongs. The categories include landslide body, fissure zone, loose accumulation area and background area. The category of each pixel is determined based on the maximum parameter value decision method and the probability. By integrating the pixels belonging to the landslide body, the region where the landslide body is located is obtained.

7. A landslide identification device, characterized in that, The device includes: The feature extraction module is used to extract features from images from multiple modalities based on the landslide body recognition model, and obtain the extracted features corresponding to each image. The extracted features are obtained through multiple feature extraction network layers. The frequency domain feature acquisition module is used to transform the extracted features to the corresponding frequency domain and perform frequency domain compression to obtain the frequency domain features corresponding to each image. The weighted feature acquisition module is used to convert the frequency domain features into weighting factors, and obtain the weighted features corresponding to each image based on the weighting factors and the extracted features corresponding to the same image. The feature stitching module is used to stitch together the weighted features corresponding to each image to obtain the fused features; The splicing feature processing module is used to process the fused features through a multi-layer dense network and to perform multiple interpolations on the processed features using bilinear interpolation. The landslide identification module is used to obtain the category of each pixel in the features after multiple interpolations using the maximum parameter value decision method, and to determine the landslide.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images