Landslide hidden danger associated element identification method and device based on multi-modal fusion and pruning

Abstract

The invention discloses a landslide hidden danger associated element identification method and device based on multi-modal fusion and pruning, and relates to the technical field of geological disaster identification. The landslide hidden danger associated element identification method based on multi-modal fusion and pruning comprises the following steps: acquiring initial data, wherein the initial data comprises aviation optical atlas data, InSAR deformation data and topographic data; carrying out waveband level superposition on the initial data to generate a multi-channel fusion data matrix; performing channel pruning, network reconstruction and fine tuning learning processing on the first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network; and obtaining to-be-identified data, inputting the to-be-identified data into the target lightweight model network, and outputting landslide hidden danger associated element data. According to the invention, multi-modal data fusion and lightweight pruning technologies can be combined, and the identification precision and real-time processing capability of landslide hidden danger associated elements are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of geological disaster identification, and more specifically, to a method and device for identifying landslide hazard-related elements using multimodal fusion and pruning. Background Art

[0002] The formation process of geological hazards is influenced by the hazard-prone environment, inducing factors, and the patterns of hazard bearing. These complex features appear in remote sensing images, making intelligent interpretation challenging. Existing intelligent identification of factors associated with landslide hazards relies primarily on multi-source data, including InSAR data, optical imaging data, and terrain data, optimizing the architecture of classic deep learning networks to automatically identify these factors.

[0003] With the development of earth observation technology, a huge amount of remote sensing images and data has been generated, which puts higher demands on the rapid extraction of regional or large-scale geological hazards. Currently, with the increase of multimodal input data, the complexity of the model has increased significantly, and the number of network parameters has expanded, making the training process more dependent on computing resources and consuming more hardware resources, which in turn has led to the low processing efficiency of intelligent identification of landslide hazard related factors. Specifically, the current intelligent identification technology of geological hazard hazards faces the following bottlenecks: (1) When directly using traditional natural language general models to process multimodal data, the extraction and utilization of data synergistic gain information is insufficient; (2) In large-scale geological hazard monitoring scenarios, there are problems such as model parameter redundancy, imbalance in model inference effect and speed. Summary of the Invention

[0004] The embodiments of the present invention provide a method and device for identifying landslide hazard-related elements by multimodal fusion and pruning, which combines multimodal data fusion with lightweight pruning technology to improve the recognition accuracy and real-time processing capability of landslide hazard-related elements.

[0005] According to the first aspect of the present invention, an embodiment of the present invention provides a method for identifying landslide hazard-related elements by multimodal fusion and pruning, the method comprising: obtaining initial data, the initial data comprising: aerial optical map data, InSAR deformation data, and terrain data; performing band-level superposition on the initial data to generate a multi-channel fusion data matrix; performing channel pruning, network reconstruction, and fine-tuning learning processing on a first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network; obtaining data to be identified and inputting the data to be identified into the target lightweight model network, and outputting landslide hazard-related element data.

[0006] The above-mentioned embodiment of the present invention generates a target lightweight model network by fusing multi-source remote sensing initial data, and performing channel pruning and other processing on the first lightweight model network based on the fused data, and then inputting the data to be identified into the target lightweight model network to obtain landslide hazard related element data. It can comprehensively apply multi-source remote sensing data and prune the redundant feature layers within the model based on the multimodal data fusion network model, significantly reducing the number of model parameters and improving the reasoning efficiency of the model's business application.

[0007] In some embodiments of the present invention, the multimodal fusion and pruning landslide hazard associated element identification method further includes: aligning initial data of different modalities, unifying spatial resolution and pixel size, and then performing band-level superposition.

[0008] In some embodiments of the present invention, the first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding part of the U-shaped structure adopts an inverse convolution and inverted residual structure.

[0009] In some embodiments of the present invention, performing channel pruning, network reconstruction, and fine-tuning learning on the first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network includes: extracting a feature map of the multi-channel fusion data matrix through the first lightweight model network; performing normalization preprocessing on the input feature map through a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value through a learnable first adjustment factor and a second adjustment factor; applying an L1 regularization constraint to the first adjustment factor of the batch normalization layer, and synchronously optimizing the network weight and the first adjustment factor. A second lightweight model network is obtained that makes the first adjustment factor sparsely distributed; a preset cropping ratio and the absolute value distribution of the first adjustment factor of each channel in the batch normalization layer are obtained, and the quantile threshold of the absolute value distribution is calculated according to the preset cropping ratio; the channels of each convolutional layer in the second lightweight model network are traversed, and the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding in-out connections and weights are removed to obtain a pruned third lightweight model network; the third lightweight model network is fine-tuned and learned to obtain a target lightweight model network.

[0010] In some embodiments of the present invention, performing normalization preprocessing on the input feature map by a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value by using a learnable first adjustment factor and a second adjustment factor includes: mapping the parameters of the input feature map to a normal distribution range with a mean of 0 and a variance of 1 by using the following formula, and performing a linear transformation on the normalized value by using the learnable first adjustment factor and the second adjustment factor:

[0011]

[0012] a o =γ·τ i +β

[0013] Among them, a i is the input feature map, μ i and σ i are the mean and standard deviation obtained from the statistics of the current batch data i, τ i is the normalized activation value, ∈ is a very small constant, γ and β are the first and second adjustment factors corresponding to the independent learning of each channel feature map, a o is the output feature map.

[0014] In some embodiments of the present invention, an L1 regularization constraint is applied to the first adjustment factor of the batch normalization layer by the following formula:

[0015]

[0016] Where (x, y) is the training input and target, W is the training weight, ∑ (x,y) l(f((x,W)),y) is the normal training loss function of the convolutional neural network, λ∑ γ∈Γ g(γ) is the L1 regularization of γ, where γ is the scaling factor, λ is the penalty coefficient, and L is the total loss function for the entire training process.

[0017] According to the second aspect of the present invention, an embodiment of the present invention provides a landslide hazard associated element identification device with multimodal fusion and pruning, the device comprising: a fusion data generation module for acquiring initial data and performing band-level superposition on the initial data to generate a multi-channel fusion data matrix, wherein the initial data comprises: aerial optical map data, InSAR deformation data, and terrain data; a lightweight model network generation module for performing channel pruning, network reconstruction, and fine-tuning learning processing on a first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network; a landslide hazard associated element identification module for acquiring data to be identified and inputting the data to be identified into the target lightweight model network, and outputting landslide hazard associated element data.

[0018] The above-mentioned embodiment of the present invention generates a target lightweight model network by fusing multi-source remote sensing initial data, and performing channel pruning and other processing on the first lightweight model network based on the fused data, and then inputting the data to be identified into the target lightweight model network to obtain landslide hazard related element data. It can comprehensively apply multi-source remote sensing data and prune the redundant feature layers within the model based on the multimodal data fusion network model, significantly reducing the number of model parameters and improving the reasoning efficiency of the model's business application.

[0019] In some embodiments of the present invention, the fusion data generation module is further used to align initial data of different modalities, unify spatial resolution and pixel size, and then perform band-level superposition.

[0020] In some embodiments of the present invention, the first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding part of the U-shaped structure adopts an inverse convolution and inverted residual structure.

[0021] In some embodiments of the present invention, performing channel pruning, network reconstruction, and fine-tuning learning on the first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network includes: extracting a feature map of the multi-channel fusion data matrix through the first lightweight model network; performing normalization preprocessing on the input feature map through a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value through a learnable first adjustment factor and a second adjustment factor; applying an L1 regularization constraint to the first adjustment factor of the batch normalization layer, and synchronously optimizing the network weight and the first adjustment factor. A second lightweight model network is obtained that makes the first adjustment factor sparsely distributed; a preset cropping ratio and the absolute value distribution of the first adjustment factor of each channel in the batch normalization layer are obtained, and the quantile threshold of the absolute value distribution is calculated according to the preset cropping ratio; the channels of each convolutional layer in the second lightweight model network are traversed, and the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding in-out connections and weights are removed to obtain a pruned third lightweight model network; the third lightweight model network is fine-tuned and learned to obtain a target lightweight model network.

[0022] In summary, the embodiments of the present invention provide a method and device for identifying landslide hazard-related elements through multimodal fusion and pruning. By comprehensively applying multi-source remote sensing data and pruning the redundant feature layers within the model based on the multimodal data fusion network model, the method significantly reduces the number of model parameters and improves the reasoning efficiency of the model's business applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 1 is a flow chart of a method for identifying landslide hazard-related elements by multimodal fusion and pruning according to an embodiment of the present invention;

[0024] Figure 2 1. It is a schematic diagram of a network structure for identifying associated elements of landslide hazards according to an embodiment of the present invention;

[0025] Figure 3 1 is a schematic diagram of the architecture of a device for identifying landslide hazard-related elements using multimodal fusion and pruning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The various aspects of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that the various embodiments described below are for illustration only and are not intended to limit the scope of protection of the present invention. Various combinations of the various embodiments of the present invention may be made without departing from the essence of the present invention.

[0027] The following is a brief explanation of the terms used below.

[0028] InSAR: Interferometric Synthetic Aperture Radar, synthetic aperture radar interferometry, refers to the interferometry technology using synthetic aperture radar.

[0029] MobileNetV2: A lightweight convolutional neural network architecture.

[0030] Figure 1 It is a flowchart of a method for identifying landslide hazard-related elements by multimodal fusion and pruning according to an embodiment of the present invention.

[0031] like Figure 1 As shown, in one embodiment of the present invention, the method for identifying landslide hazard associated elements may at least include the following steps S11, S12, S13 and S14, which are described in detail below.

[0032] In step S11, initial data is acquired, and the initial data includes: aerial optical atlas data, InSAR deformation data, and terrain data.

[0033] In step S12, the initial data is band-level superimposed to generate a multi-channel fusion data matrix. In this embodiment, the initial data of different modalities are registered, and the spatial resolution and pixel size are unified before band-level superposition is performed.

[0034] In step S13, the first lightweight model network is subjected to channel pruning, network reconstruction, and fine-tuning based on the multi-channel fusion data matrix to obtain a target lightweight model network. In this embodiment, the first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding portion of the U-shaped structure adopts a deconvolution and inverted residual structure.

[0035] In some embodiments, the target lightweight model network is obtained after channel pruning, network reconstruction, and fine-tuning learning processing of the first lightweight model network based on the multi-channel fusion data matrix, including: extracting feature maps of the multi-channel fusion data matrix through the first lightweight model network; performing normalization preprocessing on the input feature maps through the batch normalization layer in the first lightweight model network, and performing linear transformation on the normalized values ​​through the learnable first adjustment factor and the second adjustment factor; applying L1 regularization constraint to the first adjustment factor of the batch normalization layer, and synchronously optimizing the network weights and the first adjustment factor to obtain A second lightweight model network is provided in which the first adjustment factor is sparsely distributed; a preset cropping ratio and the absolute value distribution of the first adjustment factor of each channel in the batch normalization layer are obtained, and the quantile threshold of the absolute value distribution is calculated according to the preset cropping ratio; the channels of each convolutional layer in the second lightweight model network are traversed, and the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding in-out connections and weights are removed to obtain a pruned third lightweight model network; the third lightweight model network is fine-tuned and learned to obtain a target lightweight model network.

[0036] In a further embodiment, performing normalization preprocessing on the input feature map by a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value by a learnable first adjustment factor and a second adjustment factor includes:

[0037] The parameters of the input feature map are mapped to a normal distribution with a mean of 0 and a variance of 1 by the following formulas (1) and (2), and the normalized values ​​are linearly transformed by the learnable first adjustment factor and the second adjustment factor:

[0038]

[0039] a o =γ·τ i +β (2)

[0040] Among them, a i is the input feature map, μ i and σ i are the mean and standard deviation obtained from the statistics of the current batch data i, τ i is the normalized activation value, ∈ is a very small constant, γ and β are the first and second adjustment factors corresponding to the independent learning of each channel feature map, a o is the output feature map, where the first adjustment factor is a scaling factor and the second adjustment factor is an offset factor.

[0041] The L1 regularization constraint is applied to the first adjustment factor of the batch normalization layer by the following formula (3):

[0042] L=∑ (x,y) l(f((x,W)),y)+λ∑ γ∈Γ g(γ) (3)

[0043] Where (x, y) is the training input and target, W is the training weight, ∑ (x,y) l(f((x,W)),y) is the normal training loss function of the convolutional neural network, λ∑ γ∈Γ g(γ) is the L1 regularization of γ, where γ is the scaling factor (i.e., the first adjustment factor), λ is the penalty coefficient, and L represents the total loss function of the entire training process.

[0044] In step S14, the data to be identified is obtained and input into the target lightweight model network, outputting data related to landslide hazards. In this embodiment, the data to be identified is seven channels of test data, and the outputted data related to hazards includes, but is not limited to, deformation characteristics, map characteristics, or hazard-bearing body elements of the landslide hazard.

[0045] The above-mentioned method of the embodiment of the present invention is adopted to generate a target lightweight model network by fusing the initial multi-source remote sensing data and performing channel pruning and other processing on the first lightweight model network based on the fused data. The data to be identified is then input into the target lightweight model network to obtain landslide hazard related element data. It can comprehensively apply multi-source remote sensing data and prune the redundant feature layers within the model based on the multimodal data fusion network model, significantly reducing the number of model parameters and improving the reasoning efficiency of the model's business application.

[0046] The following is a method using Figure 1The multimodal fusion and pruning method for identifying landslide hazard related elements is shown in the following example to realize the identification of landslide hazard related elements. Figure 2 To explain in detail:

[0047] In this example, the landslide hazard associated factor identification method based on multimodal fusion and pruning may include at least the following steps:

[0048] Step 1: Comprehensively apply the aerial optical data, InSAR deformation data, and terrain data related to landslide hazards, using a parallel fusion strategy to perform band-level overlay to generate a multi-channel fusion data matrix. The aerial optical data consists of three visible light bands (red, green, and blue), the InSAR deformation data consists of three channels of deformation phase data, and the terrain data consists of one channel of hillshade data.

[0049] In this embodiment, the initial data of different modalities (such as airborne optical atlas data, InSAR deformation data, terrain data, etc.) are registered, the spatial resolution and pixel size are unified, and then band-level superposition is performed. Among them, when the resolution of the initial data is inconsistent, a resampling operation is performed to ensure that the data is unified to the same resolution. In this way, the data of different modalities can be aligned in geographic space and have the same spatial resolution and pixel size, which facilitates the subsequent band-level superposition of the initial data and the generation of a multi-channel fusion data matrix.

[0050] Step 2: Build Figure 2 The network structure for identifying factors associated with landslide hazards is shown. This network is based on a U-shaped structure. The encoding portion of the U-shaped structure is composed of the MobileNetV2 backbone network structure, while the decoding portion incorporates deconvolution and an inverted residual structure. The encoding portion of the U-shaped structure extracts features at different levels from the fused data (multi-channel fused data matrix) generated in step 1 to obtain feature maps at different levels. The decoding portion of the U-shaped structure then serially fuses the deconvolution and inverted residual structure with the feature maps at different levels obtained from the encoding portion to obtain high-level abstract semantic features.

[0051] In this embodiment of the landslide hazard associated factor identification network, the U-shaped decoding layer uses an inverse convolution and inverted residual structure to replace the original upsampling convolution layer, feature concatenation layer, and two 3×3 normal convolution layers. By introducing inverse convolution and inverted residual structures, the feature representation of associated factors is enhanced and the computational efficiency of the optimized model is optimized.

[0052] The various layer structures of the aforementioned landslide hazard related factor identification network structure are shown in Table 1 below:

[0053] Table 1

[0054]

[0055]

[0056] In Table 1, Conv2d 3×3 is a two-dimensional convolutional layer with a convolution kernel size of 3×3, Bneck×1 is a single bottleneck module; Bneck×2 is the continuous use of two bottleneck modules; Bneck×3 is the continuous use of three bottleneck modules; Bneck×7 is the continuous use of seven bottleneck modules; Bneck×4 is the continuous use of four bottleneck modules; Conv2d 1×1 is a two-dimensional convolutional layer with a convolution kernel size of 1×1; TransConv 4×4 is a transposed convolution with a convolution kernel size of 4×4, and InvertedRes is an inverted residual module.

[0057] Step 3: Based on the lightweight model network constructed in step 2, the channel-level pruning method is used to prune the unimportant feature layers. Specifically, a Batch Normalization layer (BN layer) is set to perform normalization preprocessing. All parameters of the input feature map are mapped to a normal distribution range with a mean of 0 and a variance of 1 through the above formulas (1) and (2), so that each mapped parameter learns the corresponding first adjustment factor γ and second adjustment factor β during the training process. Among them, γ can be regarded as the weight of each channel of the BN layer feature map. When r i When it approaches 0, the output channels corresponding to the feature map are all constants, that is, the channel has no contribution to subsequent operations.

[0058] Step 4: Sparse constraint training. Specifically, the L1 regularization constraint is applied to the first adjustment factor (i.e., scaling factor) γ of the Batch Normalization layer through the aforementioned formula (3), and the network weights and scaling factors are optimized simultaneously during the training process to induce the scaling factors to present a sparse distribution.

[0059] Step 5: Channel pruning. Specifically, after the channel sparse regularization training in step 4, a network model is obtained in which many scaling factors γ tend to 0. The absolute value distribution of the scaling factors γ∈Γ of all BN layers in the network model is counted, and the quantile threshold of the absolute value distribution is calculated according to the set pruning percentage.

[0060] Step 6: Network Reconstruction. Specifically, traverse the channels of each convolutional layer in the network obtained in Step 5, remove the channels that do not meet the aforementioned quantile threshold, and simultaneously delete the corresponding inbound and outbound connections and corresponding weights of the removed channels to obtain a compact pruned network. For example, the comparison of the number of channels of some convolutional layers before and after pruning 50% is shown in Table 2 below:

[0061] Table 2

[0062]

[0063]

[0064]

[0065] The layer index is the number that identifies each layer in the neural network, the total number of channels is the original number of channels in a layer before pruning, and the remaining number of channels is the number of channels that remain after pruning. For example, the last row in Table 1 shows that the original number of channels in the 230th layer of the neural network before pruning was 192, and the remaining number of channels is 98.

[0066] Step 7: Fine-tune the pruned network using fine-tuning techniques to ultimately obtain a more compact target model. For example, the volume, parameters, and FLOPs of the original model and the target model after 50% pruning are shown in Table 3 below:

[0067] Table 3

[0068]

[0069] Among them, Baselines is the baseline model, that is, the original model used as a comparison benchmark (the initial version that has not been pruned or optimized); γ-Prune 50% refers to the target model pruned by 50% based on the γ parameter. As shown in Table 3, the volume, number of parameters and computational complexity of the target model are significantly reduced, and the precision and accuracy of the target model are improved. It should be understood that the pruning ratio used for channel pruning in step 5 ranges from 0.01 to 0.99. The selection of the pruning ratio needs to find a balance between model volume, computational efficiency and computational performance. The final pruning ratio is usually determined through experimental verification. For example, a progressive adjustment strategy can be adopted to gradually increase the pruning ratio and simultaneously evaluate the model accuracy loss. Finally, the optimal pruning ratio that takes into account both lightweight and performance is determined based on experimental data.

[0070] Step 8: Obtain the data to be identified and input the data to be identified into the target model network, and output the landslide hazard associated element data.

[0071] The above-mentioned multimodal fusion and pruning method for identifying landslide hazard-related factors according to the embodiment of the present invention comprehensively applies multi-source remote sensing data, adopts serial and parallel interactive fusion strategies, and constructs a multimodal data fusion network model. Furthermore, the channel pruning method is used to prune the redundant feature layers within the model, which can significantly reduce the number of model parameters, improve the efficiency of model business application reasoning, and enhance the identification accuracy and real-time processing capability of landslide hazard-related factors.

[0072] Figure 3 1 is a schematic diagram of the architecture of a device for identifying landslide hazard-related elements using multimodal fusion and pruning according to an embodiment of the present invention.

[0073] like Figure 3 As shown, the device for identifying elements associated with landslide hazards includes: a fusion data generation module 310 , a lightweight model network generation module 320 , and a landslide hazard associated element identification module 330 .

[0074] The fusion data generation module 310 is used to obtain initial data and perform band-level superposition on the initial data to generate a multi-channel fusion data matrix, wherein the initial data includes: aerial optical atlas data, InSAR deformation data, and terrain data.

[0075] In this embodiment, the fusion data acquisition module is further used to align initial data of different modalities, unify spatial resolution and pixel size, and then perform band-level superposition.

[0076] The lightweight model network generation module 320 is used to perform channel pruning, network reconstruction, and fine-tuning learning processing on the first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network.

[0077] In this embodiment, the first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding part of the U-shaped structure adopts an inverse convolution and inverted residual structure.

[0078] In some embodiments, the target lightweight model network is obtained after channel pruning, network reconstruction, and fine-tuning learning processing of the first lightweight model network based on the multi-channel fusion data matrix, including: extracting feature maps of the multi-channel fusion data matrix through the first lightweight model network; performing normalization preprocessing on the input feature maps through the batch normalization layer in the first lightweight model network, and performing linear transformation on the normalized values ​​through the learnable first adjustment factor and the second adjustment factor; applying L1 regularization constraint to the first adjustment factor of the batch normalization layer, and synchronously optimizing the network weights and the first adjustment factor to obtain A second lightweight model network is provided in which the first adjustment factor is sparsely distributed; a preset cropping ratio and the absolute value distribution of the first adjustment factor of each channel in the batch normalization layer are obtained, and the quantile threshold of the absolute value distribution is calculated according to the preset cropping ratio; the channels of each convolutional layer in the second lightweight model network are traversed, and the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding in-out connections and weights are removed to obtain a pruned third lightweight model network; the third lightweight model network is fine-tuned and learned to obtain a target lightweight model network.

[0079] The landslide hazard associated element identification module 330 is used to obtain data to be identified and input the data to be identified into the target lightweight model network, and output landslide hazard associated element data.

[0080] The above-mentioned device of the embodiment of the present invention fuses the multi-source remote sensing initial data, and generates a target lightweight model network after performing channel pruning and other processing on the first lightweight model network based on the fused data. The data to be identified is then input into the target lightweight model network to obtain landslide hazard related element data. It can comprehensively apply multi-source remote sensing data and prune the redundant feature layers within the model based on the multimodal data fusion network model, significantly reducing the number of model parameters and improving the reasoning efficiency of the model's business application.

[0081] Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on this understanding, all or part of the contribution of the technical solution of the present invention to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0082] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium having computer-readable instructions or programs stored thereon. When the computer-readable instructions or programs are executed by a processor, the processor is caused to perform the following operations: the operations include the steps included in the method described in any one of the above embodiments, which will not be repeated here. The storage medium may include, for example, a CD, a hard disk, a floppy disk, a flash memory, a magnetic tape, etc. Part or all of the above embodiments of the present invention may be implemented by a computer device, which includes the above storage medium and a processor, and the processor executes the computer-readable instructions in the storage medium to perform the steps, operations, and processing described in any one of the above embodiments.

[0083] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may be modified or some of the technical features thereof may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for identifying landslide hazard related factors using multimodal fusion and pruning, the method comprising: Acquiring initial data, the initial data including: aerial optical atlas data, InSAR deformation data, and terrain data; Performing band-level superposition on the initial data to generate a multi-channel fusion data matrix; Based on the multi-channel fusion data matrix, the first lightweight model network is subjected to channel pruning, network reconstruction, and fine-tuning learning processing to obtain a target lightweight model network; The data to be identified is obtained and input into the target lightweight model network, and the landslide hazard associated element data is output.

2. The method according to claim 1, wherein The method further includes: aligning initial data of different modalities, unifying spatial resolution and pixel size, and then performing band-level superposition.

3. The method according to claim 1 or 2, wherein: The first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding part of the U-shaped structure adopts an inverse convolution and inverted residual structure.

4. The method according to claim 3, wherein The target lightweight model network is obtained by performing channel pruning, network reconstruction, and fine-tuning learning on the first lightweight model network based on the multi-channel fusion data matrix, including: Extracting feature maps from the multi-channel fusion data matrix through the first lightweight model network; Performing normalization preprocessing on the input feature map through a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value through a learnable first adjustment factor and a second adjustment factor; Applying an L1 regularization constraint to the first adjustment factor of the batch normalization layer, and simultaneously optimizing the network weights and the first adjustment factor to obtain a second lightweight model network in which the first adjustment factor is sparsely distributed; Obtaining a preset cropping ratio and an absolute value distribution of a first adjustment factor of each channel in the batch normalization layer, and calculating a quantile threshold of the absolute value distribution according to the preset cropping ratio; Traversing the channels of each convolutional layer in the second lightweight model network, and removing the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding inbound and outbound connections and weights, to obtain a pruned third lightweight model network; Fine-tune the third lightweight model network to obtain a target lightweight model network.

5. The method according to claim 4, wherein Performing normalization preprocessing on the input feature map through a batch normalization layer in the first lightweight model network, and performing linear transformation on the normalized value through a learnable first adjustment factor and a second adjustment factor includes: The parameters of the input feature map are mapped to a normal distribution with a mean of 0 and a variance of 1 using the following formula. The normalized values ​​are linearly transformed using the learnable first and second adjustment factors to restore the expressive power of the data: a o =γ·τ i +b Among them, a i is the input feature map, μ i and σ i are the mean and standard deviation obtained from the statistics of the current batch data i, τ i is the normalized activation value, ∈ is a very small constant, γ and β are the first and second adjustment factors corresponding to the independent learning of each channel feature map, a o is the output feature map.

6. The method according to claim 4, wherein An L1 regularization constraint is applied to the first adjustment factor of the batch normalization layer using the following formula: Where (x, y) is the training input and target, W is the training weight, ∑ (x,y) l(f((x,W)),y) is the normal training loss function of the convolutional neural network, λ∑ γ∈Γ g(γ) is the L1 regularization of γ, where γ is the first adjustment factor, λ is the penalty coefficient, and L is the total loss function of the entire training process.

7. A device for identifying landslide hazard related factors based on multimodal fusion and pruning, characterized in that: The device comprises: A fusion data generation module is used to obtain initial data and perform band-level superposition on the initial data to generate a multi-channel fusion data matrix, wherein the initial data includes: aerial optical atlas data, InSAR deformation data, and terrain data; A lightweight model network generation module is used to perform channel pruning, network reconstruction, and fine-tuning learning on the first lightweight model network based on the multi-channel fusion data matrix to obtain a target lightweight model network; The landslide hazard associated element identification module is used to obtain data to be identified and input the data to be identified into the target lightweight model network, and output landslide hazard associated element data.

8. The device according to claim 7, wherein The fusion data generation module is also used to align the initial data of different modalities, unify the spatial resolution and pixel size, and then perform band-level superposition.

9. The device according to claim 7 or 8, characterized in that The first lightweight model network is a model network based on a U-shaped structure, wherein the encoder of the U-shaped structure adopts a partial network structure of MobileNetV2, and the decoding part of the U-shaped structure adopts an inverse convolution and inverted residual structure.

10. The device according to claim 9, wherein The target lightweight model network is obtained by performing channel pruning, network reconstruction, and fine-tuning learning on the first lightweight model network based on the multi-channel fusion data matrix, including: Extracting feature maps from the multi-channel fusion data matrix through the first lightweight model network; Performing normalization preprocessing on the input feature map through a batch normalization layer in the first lightweight model network, and performing a linear transformation on the normalized value through a learnable first adjustment factor and a second adjustment factor; Applying an L1 regularization constraint to the first adjustment factor of the batch normalization layer, and simultaneously optimizing the network weights and the first adjustment factor to obtain a second lightweight model network in which the first adjustment factor is sparsely distributed; Obtaining a preset cropping ratio and an absolute value distribution of a first adjustment factor of each channel in the batch normalization layer, and calculating a quantile threshold of the absolute value distribution according to the preset cropping ratio; Traversing the channels of each convolutional layer in the second lightweight model network, and removing the channels whose absolute values ​​of the first adjustment factors corresponding to the channels of each convolutional layer in the second lightweight model network are less than the quantile threshold and their corresponding inbound and outbound connections and weights, to obtain a pruned third lightweight model network; Fine-tune the third lightweight model network to obtain a target lightweight model network.

Images