Intelligent creeping type landslide hidden danger identification method combining image processing and semantic understanding

Abstract

The invention discloses a creep type landslide hidden danger intelligent identification method combining image processing and semantic understanding, and the method comprises the steps: firstly, employing a high-precision geometric modeling and filtering technology to inhibit irrelevant phases in an InSAR interferogram, and keeping the deformation information related to a landslide; secondly, enhancing signal features through phase gradient, RGB channel mapping and a generative adversarial network, and improving the characterization capability of the signals; and then, a deep learning model is optimized by using convolution operation, a self-attention mechanism and structural re-parameterization, and the learning ability of the landslide hidden danger is improved. In the feature coding stage, vision and language features are obtained through a multi-mode subtitle generation model and mapped to the same embedding space to achieve seamless connection. And finally, performing problem prediction on the landslide hidden danger by utilizing an automatic regression language generation model and a multi-layer perceptron. According to the scheme, by integrating vision, language and deep learning technologies, the landslide hidden danger monitoring efficiency and accuracy are remarkably improved, and effective support is provided for disaster response and management.

Description

Technical Field

[0001] The present invention belongs to the technical field of synthetic aperture radar, and in particular relates to an intelligent identification method for creep-type landslide hazards by combining image processing with semantic understanding. Background Art

[0002] Landslide is a common and destructive geological disaster, which refers to the downward and outward sliding movement of rock and soil along a specific sliding surface under the action of gravity. There are two main types of landslide hazards: one is the area where landslide has not yet occurred, but has continued to deform under the action of external forces, and may even slide instability (i.e., potential landslide area); the other is the area where landslide has occurred before. Although the landslide body has become stable, there is a potential risk of deformation or recurrence of sliding due to changes in external conditions. The occurrence of landslides not only leads to significant changes in the surface morphology, but also has a serious impact on human society and economic activities. Therefore, it is of great significance to accurately identify and assess landslide hazards. Traditional risk identification methods mainly rely on field surveys and field data collection. However, these methods have many limitations. First, the spatial coverage of field surveys is usually limited, making it difficult to comprehensively monitor large-scale landslides. Second, the lack of real-time data collection on the spot causes the data to lag behind geological changes, thereby weakening its effectiveness in disaster prevention and mitigation. In addition, field surveys often require a lot of manpower and material resources, which is costly.

[0003] The rapid development of remote sensing technology has led to a profound change in the way landslide hazards are identified. Remote sensing technology enables landslide monitoring to be carried out on a large scale and in multiple dimensions, significantly improving the efficiency and accuracy of data collection. The non-contact nature of remote sensing images not only avoids the manpower and time costs of traditional field surveys, but also covers a wide area, allowing geological disaster monitoring in remote or difficult-to-reach areas. However, the effectiveness of optical remote sensing images is affected by many factors, such as sunshine conditions, weather changes, cloud cover, and vegetation cover. In comparison, synthetic aperture radar (SAR) data presents unique advantages. First, SAR has all-weather and all-day imaging capabilities, and can penetrate clouds and precipitation, so it can still provide high-quality image data in emergency scenarios such as severe weather. In addition, SAR uses active imaging technology, and its strong penetration enables it to effectively obtain more realistic surface information.

[0004] Interferometric synthetic aperture radar (InSAR) is a remote sensing technology based on SAR data. It can accurately measure surface deformation and elevation by calculating phase difference. InSAR technology can capture small displacement changes of the surface. Its high sensitivity to these changes makes it of great application value in landslide deformation monitoring. Differential InSAR (DInSAR) is affected by the quality of the elevation model and has the problem of temporal and spatial decoherence. Therefore, most studies use time-series InSAR technology. At present, a variety of time-series InSAR methods have been widely used in the monitoring of landslide activities. However, due to the diversity of landslide hazard characteristics and the complex geomorphic environment, the current interpretation of InSAR deformation products still requires visual judgment through human-computer interaction, which makes the accuracy of the results largely depend on expert knowledge.

[0005] With its powerful feature extraction and pattern recognition capabilities, deep learning is profoundly changing InSAR monitoring technology and its application in disaster scenarios such as earthquakes, volcanoes, and glaciers. Convolutional neural networks (CNNs) are particularly outstanding in spatial feature extraction, and can identify key edge, texture, and shape information from remote sensing images and accurately capture subtle changes. Transformer models, with their long-distance dependency processing capabilities, effectively associate various time nodes in the time dimension and complete sequence prediction. At the same time, they break the receptive field limitations in space, making the network more globally focused. As an emerging architecture, the Mamba model is based on the selective state space model (SSM), which reduces computational overhead while achieving efficient reasoning and real-time response, and is particularly suitable for the timeliness requirements of disaster monitoring.

[0006] In the study of intelligent identification of landslide hazards, the combination of deep learning and InSAR technology has experienced a transition from patch-level or pixel-level regional positioning and sensitivity mapping to more fine-grained precise positioning and boundary identification technology. At present, the research usually uses deformation products as network input, and uses semantic segmentation and target detection technology to accurately delineate the boundaries of landslide areas. Although deformation products have shown significant potential in intelligent identification of landslides, there are still many technical difficulties in practical applications. First, in low-coherence and complex terrain areas, unwrapping errors are prone to occur. The existing automatic unwrapping algorithms are still difficult to completely solve these problems, resulting in the gradual accumulation of noise phases in the deformation results, increasing the uncertainty of identification. In addition, the data processing after unwrapping requires comprehensive analysis of multi-dimensional information such as time, space and deformation amplitude, and usually relies on expert knowledge for parameter tuning. Secondly, there is a lack of effective self-interpretation and self-analysis mechanisms. Subsequent analysis and interpretation work still needs to be combined with rich geological or disaster knowledge to identify the types of landslide hazard areas identified by visual models, analyze disaster mechanisms and conduct risk assessment.

[0007] In summary, although the current research on intelligent identification of landslide hazards has made certain progress, there are still significant limitations. First, the high dependence of identification results on data quality is still a key issue. Current research generally uses unwrapped products as model input, but this method is susceptible to phase unwrapping errors and error accumulation, thereby reducing the accuracy of identification. Secondly, the InSAR processing flow contains a series of complex and professional steps, such as parameter optimization, error correction, etc., which require reliance on expert knowledge. At the same time, in practical applications, due to the lack of relevant professional background of users, this may lead to deviations in data analysis and result interpretation. Therefore, the present invention focuses on reducing the dependence on high-quality data and expert knowledge to improve the accuracy and wide applicability of intelligent identification of landslide hazards. Summary of the invention

[0008] The present invention aims to solve the technical problems existing in the background technology and to provide an intelligent identification method for creep-type landslide hazards by combining image processing and semantic understanding, so as to solve the high dependence on high-quality data and expert knowledge in the current landslide hazard identification process. The detection, analysis and feedback of landslide hazards will be realized in an intelligent and automated manner, which will not only improve the recognition accuracy, but also enhance the monitoring and emergency response capabilities of landslide risks, and provide innovative technical support for the prevention, monitoring and early warning of landslide disasters.

[0009] In order to solve the technical problem, the technical solution of the present invention is:

[0010] An intelligent identification method for creep landslide hazards combining image processing and semantic understanding, the method comprising:

[0011] S1: Input the winding InSAR interferogram; first, through geometric modeling, phase filtering and the introduction of external data, the image is processed to suppress the irrelevant phase caused by the earth's ellipsoid, terrain, atmosphere and Doppler frequency shift; then, the deformation signal of landslide hazard is highlighted by calculating the deformation phase gradient, signal trigonometric function decomposition and RGB three-color channel mapping; finally, the deep learning semantic segmentation method combining convolution operation, self-attention mechanism, structural reparameterization and selective state space model technology is used to identify the landslide hazard in the target area; output the raster and vector data of the landslide hazard area identified by the deep learning semantic segmentation method;

[0012] S2: Input the winding InSAR interferogram and the landslide hazard area segmentation result produced by step S1; first, generate a text description of the landslide hazard area by defining a template and expert annotation, including landslide type, landslide disaster mechanism, and landslide state characteristics; then, generate a feature description of the landslide hazard by a deep learning image subtitle generation method combining vision and text processing; output the text description of the landslide hazard area interpreted by the deep learning image subtitle generation method;

[0013] S3: Input the landslide hazard area segmentation result produced by step S1 and the text description produced by step S2; first, generate a landslide hazard area question-answer dataset by defining templates and expert annotations, and the question forms include: "Will the landslide hazard slip?", "Is it necessary to evacuate people?", "What emergency measures need to be taken?", etc.; finally, generate an interactive question-answer mechanism for the landslide hazard area through a deep learning visual question-answering mechanism method that combines vision and text processing; output the interpretation and judgment of the landslide hazard area and related issues made by the deep learning visual question-answering mechanism method.

[0014] Further, the step S1 comprises:

[0015] S101: Through high-precision geometric modeling, filtering and external data correction, irrelevant phase interference in the interference pattern is suppressed, the pure interference phase information related to landslide hazards is retained, and the filtered and corrected interference pattern is generated to provide high-quality input data for subsequent deformation signal enhancement and learning;

[0016] S102: Generate RGB images or other high-dimensional feature data containing enhanced landslide hazard deformation features from InSAR data through phase gradient enhancement, RGB channel mapping and generative adversarial network methods, providing easy-to-learn and identifiable landslide deformation signal representation for semantic segmentation networks;

[0017] S103: By integrating convolution operations, self-attention mechanism and structural reparameterization methods, the training and reasoning structures are optimized. The optimized deep learning model has a segmentation network with accurate identification of landslide hazard areas and efficient reasoning capabilities, and outputs accurate segmentation results of landslide hazards.

[0018] Further, the step S2 comprises:

[0019] S201: Create a dataset of matching landslide images, i.e., interference maps and segmentation results, and their corresponding text descriptions to support multimodal learning of the IC model; first, by defining description templates and annotating them by geological experts to generate text descriptions, second, matching images and texts and performing diversity checks to form visual-language pairs, and finally, by augmenting the visual data and text data separately to prevent potential overfitting risks during training;

[0020] S202: By constructing a multimodal subtitle generation model, the landslide hazard segmentation results and interference phases are encoded into visual feature vectors using a visual encoder, and the description text is mapped into a language feature vector using a language encoder, thereby generating a landslide feature description, including visual feature vectors from the landslide hazard segmentation results and interference phases, and language feature vectors from the description text, providing basic data for subsequent feature alignment and language generation; by constructing a shared feature space, the visual feature vectors and language feature vectors are mapped to the same embedding space to maximize the similarity of matching image and text pairs, and a lightweight mapping network is combined to generate an embedding vector to form a seamless connection from vision to language; by inputting the connection sequence into the autoregressive language generation model, the landslide description is predicted under the prefix condition, so that the cross entropy loss function is used to train the generation component to optimize the prediction effect, and finally an accurate text description of the landslide hazard is output.

[0021] Further, the step S3 comprises:

[0022] S301: Create a dataset of matching landslide images and their corresponding interpretation text descriptions, as well as question-answering text descriptions, to support multimodal learning of the VQA model. Specifically, first, define description templates and generate text descriptions through annotation by geological experts. Second, match images and texts and perform diversity checks to form visual-language pairs. Finally, augment visual data and text data separately to prevent potential overfitting risks during training.

[0023] S302: First, use the ViT and BERT models to encode the features of image and text data respectively, generate form-aligned visual feature vectors and language feature vectors, and form a preliminary embedding representation; second, use the multimodal fusion method to make the visual feature vector and the language feature vector mutually weighted and fused to capture subtle differences in information; finally, input the fused embedding into the multi-layer perceptron to achieve accurate prediction of the problem, and enhance the model's adaptability to different types of problems through multi-task learning.

[0024] Further, the step S1 specifically includes:

[0025] S101: Suppressing irrelevant phase of the interferogram:

[0026] For irrelevant phase interference in the winding phase image; first, the interference phase caused by the reference ellipsoid and terrain is characterized and removed by high-precision geometric modeling; second, for the noise phase, before interference, the phase caused by Doppler frequency shift is filtered in azimuth, and the speckle noise caused by the coherent superposition of backscattered waves from multiple scatterers is processed by multi-look processing. After interference, the data is further smoothed by Goldstein frequency domain filtering based on statistical characteristics. For the atmospheric delay phase caused by the atmospheric phase screen, it is simulated and removed by meteorological data and GNSS data;

[0027] In SAR images, the complex signal of each pixel can be expressed based on the Euler formula: S = A·e jφ , where e jφ =cos(φ)+jsin(φ), A represents the amplitude, represents the phase, R represents the propagation path, λ represents the wavelength of the radar signal, j is the imaginary unit, and for two interferometric SAR images, they are expressed in complex form:

[0028]

[0029] in, S 1 represents the first SAR image, S 2 represents the second SAR image. By multiplying the complex conjugate of S1 and S2, we get:

[0030]

[0031] Among them, * represents complex conjugate, A1 represents the amplitude of S1, A2 represents the amplitude of S2, φ1 represents the phase of S1, φ2 represents the phase of S2, and after complex multiplication expansion, the phase difference can be expressed as:

[0032]

[0033] Among them, Δφ represents the phase difference, arg represents the phase angle, and the interference phase can be expressed according to the components as follows:

[0034]

[0035] in, represents the reference ellipsoid phase, represents the terrain phase, represents the deformation phase, represents the atmospheric delay phase, represents the random noise phase,

[0036] Without considering and In case, as well as It can be represented by signal interference modeling. It refers to the interference phase corresponding to the projection point of equal slant distance on the reference ellipsoid of any point on the ground. The observation point P1 is projected to the reference ellipsoid to P0 with the distance R1 of the main image as the radius and the velocity vector of the satellite flight as the axis. The interference phase at P0 is the interference phase at P1. It can be expressed as:

[0037]

[0038] Among them, R1 and R2 represent the distances from the first and second imaging points to P1, r1 and r2 represent the distances from the first and second imaging points to P0, B || represents the length of the horizontal baseline, B represents the length of the baseline, θ represents the incident angle, α represents the horizontal angle between the first and second imaging points, It refers to the additional interference phase caused by Δθ caused by the earth height h. The slant range difference ΔR of the two images can be expressed as: ΔR = Bsin(θ-α+Δθ) = Bsin(θ-α)cosΔθ+BsinΔθcos(θ-α). When Δθ→0, we have Therefore, it can be further simplified to: ΔR = Bsin(θ-α) + BΔθcos(θ-α), and ΔR is used to represent the interference phase, that is:

[0039]

[0040] Therefore, the terrain phase can be expressed as: in, B ⊥ Indicates the length of the vertical baseline. Because the curvature of the earth is small in a small area, Among them, H represents the geodetic height of the first imaging point, so After being brought in, the terrain phase can be further simplified as:

[0041]

[0042] SAR satellites are affected by the atmospheric phase screen during signal propagation, resulting in additional propagation paths, which in turn produce atmospheric delay phases, mainly including delays caused by the ionosphere and troposphere. The ionosphere delay is caused by changes in electron density and is inversely correlated with the signal wavelength:

[0043]

[0044] in, represents the ionospheric delayed phase, TEC represents the total electron content, and the Sentinel-1A / B radar image is used as the original data, that is, the L band. Therefore, it is assumed that Right now The tropospheric delay is dominated by the tropospheric delay phase, which consists of stratified and turbulent components. The stratified component is related to the terrain height and is removed by considering the terrain factor. The turbulent component is related to the water vapor content and is removed by introducing meteorological and GNSS data for simulation.

[0045] Interference phase filtering can be divided into pre-filtering and post-filtering according to the stage of data processing; pre-filtering is used to suppress noise caused by Doppler decorrelation and speckle effect, etc.; among them, Doppler decorrelation is manifested as spectrum shift of primary and secondary SAR images in azimuth and slant range, and this effect is weakened by pre-filtering in two directions respectively. The speckle effect refers to the speckle of multiplicative noise caused by the coherent superposition of backscattered waves generated by scatterers on the incident signal between scatterers. This noise can be suppressed by multi-look processing:

[0046]

[0047] in, represents the average intensity value after multi-view processing, I represents the intensity value of the i-th view, L represents the total number of views, and post-filtering is based on the morphological or statistical characteristics of the interference phase to further remove the noise in the interference pattern; Goldstein frequency domain filtering will be used to further remove the noise in the interference pattern:

[0048] H(u,v)=S{Z(u,v)} α Z(u,v) (10)

[0049] Where a∈[0,1] represents the filter parameter; S{·} is the smoothing operator; u and v are the spatial frequencies; Z(u,v) and H(u,v) are the Fourier spectra of the interference pattern before and after filtering, respectively;

[0050] The interference results are converted into The interference phase is dominant. Perform related enhancement operations; first, enhance the sensitivity of the data to deformation perception by means of phase gradient. Phase gradient refers to the gradient of the phase difference between adjacent pixels in the interference pattern, that is, the rate of change, which indicates the intensity and direction of local surface deformation:

[0051]

[0052] in, represents the gradient of the phase value along the direction x, represents the gradient of the phase value along the direction y, φ(x,y) represents the phase value of a pixel position (x,y) in the interference pattern, φ(x+1,y)-φ(x,y) and φ(x,y+1)-φ(x,y) represent the phase differences between adjacent pixels in the x and y directions, respectively, and Δx and Δy represent the pixel spacing along the direction x and direction y, respectively;

[0053] S102: Enhanced deformation signal:

[0054] In order to solve the problem of weak signal of landslide hazard, two different research perspectives are used to enhance it. First, from the perspective of InSAR data processing, phase gradient and trigonometric function decomposition are used to enhance the signal's ability to represent landslide hazards. Second, from the perspective of computer vision, generative adversarial networks are used to perform data augmentation at the simulation level for small sample tasks. At the same time, RGB three-color channel mapping can also be used to reduce the difficulty of network learning.

[0055] First, the ability of the data to express deformation features is enhanced by color rendering. After mapping the single-channel phase or phase gradient data to an RGB image, different color channels are used to encode the strength and direction information of the phase gradient, thereby enhancing the feature expression capability. Specifically, the interference phase is first standardized to between [0,1]:

[0056]

[0057] Among them, φ and φ norm Represent the phase before and after normalization respectively. Then, a discrete color list C = {c i} i＝1,2,...,n , where each c i =(R i ,G i ,B i ), and calculate the RGB value by linear interpolation; in the semantic segmentation task, the convolution operation in CNNs extracts the local features of the image through the convolution kernel; in the convolution operation, the input data With convolution kernel Convolution is performed, U and V represent the length and width of the input, C represents the number of input channels, H and W represent the length and width of the convolution kernel, D represents the number of output channels, and a bias matrix is ​​added R and T represent the length and width of the output, B z Represents the bias of the zth convolution to generate the output The value of its z-th channel can be expressed as:

[0058]

[0059] Among them, O (：,：,z)Represents the output value of the zth channel, M (：,：,k) represents the input value of the kth channel, * represents the convolution operation, F (：,：,k,z) Indicates that (：,：,z) and M (：,：,k) The corresponding convolution kernel, B (：,：,z) is a biased z The matrix formed is:

[0060] In the self-attention mechanism, the input sequence first undergoes three linear transformations to obtain three matrices, the query matrix Q, the key matrix K, and the value matrix V, and then calculates the attention:

[0061]

[0062] Where Attention(Q,K,V) represents the value of the self-attention mechanism, d k is the dimension of Q and K, and softmax is the activation function;

[0063] Step S103: Learning deformation signal:

[0064] The learning ability of the model is enhanced by optimizing the network training structure and the inference structure. Specifically, in the network training structure, the convolution operation and the self-attention mechanism are integrated to enhance the network's grasp of global information. Secondly, for the inference structure, the convolution blocks in the training structure are optimized using structural reparameterization and selective state space models to achieve algorithm transfer. CNNs are deeply integrated with the self-attention mechanism, and the spatial features and global dependencies are combined to enhance the detection ability of landslides in complex terrain areas. In terms of specific implementation, a multi-level feature extraction module combining CNNs and the self-attention mechanism is designed to capture the local boundary features of the landslide area through convolution operations, and then the information flow of cross-regional features is realized with the help of the self-attention mechanism. This module extracts and fuses the spatial structure features and deformation information of the landslide area layer by layer through the stacking of multiple layers of deep networks.

[0065] According to the above convolution calculation formula, convolution only involves linear operations in the entire calculation process, so it is a linear operation. For convolutions with the same structure, they are distributive and additive:

[0066] Let F'←cF, then c(M*F)=M*F' (15)

[0067] M*F 1 +M*F 2 =M*(F 1 +F 2 ), let F'←F 1 +F 2 , there is M*F 1 +M*F2 =M*F'(16)

[0068] Where M represents an arbitrary constant, F, F 1 、F 2 Represents different convolution kernels, and * represents convolution operation. Structural reparameterization is applied to the convolution operation part in the multi-level feature extraction module, so that the segmentation network can contain multiple branches, redundant convolution kernels or non-standard structures during training. In the inference stage, these redundant structures are merged into a simpler equivalent structure, thereby accelerating inference while maintaining the performance during training.

[0069] A state space is the minimum number of variables that contain a complete description of a system; it is a way to represent a problem mathematically by defining the possible states of the system; a state space model is used to describe these states and predict the next state of the system based on the inputs:

[0070]

[0071] Among them, A, B, C, and D all represent coefficient matrices, h(t) and h'(t) represent the current and next states respectively, x(t) represents the input, and y(t) represents the output. For matrix A, in order to keep a large historical memory, the high-order polynomial projection HiPPO operator is used to construct it:

[0072]

[0073] Among them, n and k are used to define the index of A; in order for SSM to receive sequence data and generate sequence output, it is necessary to replace the matrix of the calculation state with a discrete matrix:

[0074]

[0075] Where Δ represents the discretized time step, is the discretized matrix, I represents the unit matrix; at the same time, the data is selectively compressed into the state, even if the matrices B, C and Δ depend on the input, the selective SSM divides the entire time series into multiple shorter blocks, each block contains multiple consecutive time steps; within each block, the state update still maintains the recursive order to ensure the temporal dependency of the data; in the segmentation model, the selective SSM is combined with the self-attention mechanism.

[0076] Further, the step S2 specifically includes:

[0077] Step S201: Constructing a visual-language representation dataset:

[0078] Define a set of templates τ = [T1, T2, ..., T n ] Each template T iIt includes variable placeholders for inserting the characteristic information of the landslide. The template format is as follows:

[0079] T i ='Landslide type <category>, landslide characteristics <evaluation>'(20)

[0080] Among them, <type> is used to record the type of landslide, and <evaluation> is used to record the disaster mechanism and state characteristic information of the landslide; according to the interference map and segmentation results, the image features F = [f1, f2, ..., f n ], and the geological experts fill in the templates with features; then, image-text matching is performed, and the image features F and the text template τ are matched to generate a paired data set P = {(f1, T1), (f2, T2), ..., (f n ,T n )}; Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure; Finally, the dataset is augmented: for visual data, geometric transformation, color transformation and noise injection are used for augmentation; for text data, synonym replacement and variant description are used for augmentation; At this point, the final dataset is output in, and Represent the augmented visual features and text descriptions respectively;

[0081] Step S202: Learning landslide feature description:

[0082] Based on the multimodal learning framework, a multimodal caption generation model is constructed using a visual encoder, a language encoder and an attention mechanism. The visual encoder encodes the input landslide hazard segmentation result and its corresponding interference phase into a visual feature vector V. The language encoder maps the description text of the landslide hazard area to the same semantic space and encodes it into a language feature vector L:

[0083]

[0084] Among them, x i ,y i represents the value of the index at the i-th position of the (entangled) InSAR interferogram and the landslide hazard area segmentation result produced by S1, respectively, i Indicates the value of the index at the i-th position of the landslide text description; Encoder vision and Encoder language denote the visual and language encoders respectively;

[0085] By constructing a shared feature space, V and L are mapped to the same embedding space; the training goal is to maximize the cosine similarity of similar image and text pairs, while minimizing the similarity of non-matching pairs:

[0086]

[0087] Among them, ||·|| represents the Euclidean norm, Sim(V,L) represents the similarity between visual features and language features, and then a lightweight mapping network is used to map the embedding vector of the feature space into k embedding vectors:

[0088]

[0089] Among them, F(CLIP(x i )) means input x i The feature vectors generated by the CLIP model are further mapped to the function of the embedding space; each vector Has the same dimensions as word embeddings; and is concatenated with the visual embeddings:

[0090]

[0091] By concatenating the prefix with the caption sequence during training Input into the language model, so that the model predicts the token of the title in an autoregressive manner under the condition of the prefix; use cross entropy loss to train the mapping component.

[0092] Further, the step S3 specifically includes:

[0093] Step S301: Constructing a visual-language question answering dataset:

[0094] The question-answer text description includes questions Q and answers A. First, define the question template set T Q =[T Q1 ,T Q2 ,...,T Qn ], where each question template T Qi Questions are asked about the specific characteristics of the landslide; then, a set of answer templates T is defined A =[T A1 ,T A2 ,...,T An ], where each question template T Ai The corresponding features extracted from the landslide image are filled in, and the question and answer form a question-answer pair (Q, A) = (T Q (F),T A(F)); Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure; Finally, the dataset is augmented; At this point, the final dataset is output in, and Represent the augmented visual features and text (questions and corresponding answers) respectively;

[0095] Step S302: Learning landslide characteristics Answer:

[0096] A landslide hazard VQA model is designed, which includes three steps: feature extraction, fusion and prediction. In the feature extraction part, ViT and BERT models are used to extract the features of images and texts respectively. The image input is divided into fixed-size tiles by ViT and encoded into a series of embedded representations, which are defined as:

[0097]

[0098] Among them, d v represents the dimension of embedding, and K represents the number of tokens after segmentation. For a given question text input, the BERT model divides the text into several word embeddings, defined as:

[0099]

[0100] Among them, d q represents the dimension of embedding, M represents the number of tokens after segmentation; self-attention and cross-attention mechanisms are introduced to generate context-dependent embeddings; given the text representation Q generated by BERT i The attention weights are calculated through two fully connected layers and the activation function ReLU:

[0101]

[0102] in, represents the attention weight score of the mth token, and are the learned weight matrices respectively, and then, the attention weights are used to represent the text q m Perform weighted summation to get attention-weighted text embedding:

[0103]

[0104] in, Represents the attention weight score of the mth token activated by softmax, and then, for the image embedding V generated by ViT i , first combining it with the attention-weighted text representation q:

[0105]

[0106] in, and is the linear transformation matrix; then the combined image is represented by v′ k Calculate attention weights;

[0107] The image embeddings are weighted summed to obtain the attention-weighted image embedding:

[0108]

[0109] in, represents the k-th image embedding v k Attention weights, in the fusion stage, MUTAN is selected to combine image and text embeddings. MUTAN uses Tucker decomposition to fuse the tensor Decomposed into three matrices and a small kernel tensor:

[0110] T=((T c ×1W q )×2W v )×3W o (31)

[0111] in, They are the matrices for preparing the fusion process;

[0112] The fused embedding is passed into a multi-layer perceptron, with each dimension corresponding to a possible answer category. The model also integrates multi-task learning at this stage to enhance the adaptability of the model by training the attention module and classification weights for each question type separately:

[0113]

[0114] Among them, y represents the final output, MLP represents multi-layer perceptron, Represents a multimodal fusion mechanism for integrating image features And the text feature q is fused.

[0115] Compared with the prior art, the advantages of the present invention are:

[0116] The detection, analysis and feedback of landslide hazards will be realized in an intelligent and automated manner, which will not only improve the recognition accuracy, but also enhance the monitoring and emergency response capabilities of landslide risks, and provide innovative technical support for the prevention, monitoring and early warning of landslide disasters. This will solve the current high dependence on high-quality data and expert knowledge in the process of identifying landslide hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1, the overall technical roadmap of the present invention;

[0118] Figure 2 , a technical roadmap for InSAR semantic segmentation models for entangled phase processing;

[0119] Figure 3 , a technical roadmap for multimodal semantic parsing and image caption generation mechanisms;

[0120] Figure 4 , technical roadmap for intelligent feedback and visual question-answering mechanism. DETAILED DESCRIPTION

[0121] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0122] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0123] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0124] Embodiment 1:

[0125] The key technology focused on in this invention is to develop automation, intelligence and multi-modal interpretation around landslide hazard identification, striving to overcome the high dependence of traditional methods on data quality and expert experience, and solve the bottleneck problem in landslide disaster monitoring. The specific key technologies are as follows:

[0126] Discovering landslide hazards from interference patterns. In the identification of landslide hazards, the phase unwrapping process of InSAR data often brings errors and operational complexity. This paper reduces the dependence of traditional phase unwrapping and time series processing on data quality by designing a semantic segmentation model directly based on the entangled phase, thereby improving the adaptability of the model in low coherence and complex terrain areas. Key technologies include:

[0127] 1) Suppression of irrelevant phase in interference pattern. In view of irrelevant phase interference in the entangled phase image, firstly, the present invention will characterize and remove the interference phase caused by the reference ellipsoid and terrain by means of high-precision geometric modeling. Secondly, for the noise phase, before interference, the phase caused by the Doppler frequency shift is filtered in the azimuth (range) direction, and the speckle noise caused by the coherent superposition of backscattered waves from multiple scatterers is processed by multi-view processing. After interference, the data is further smoothed by using Goldstein frequency domain filtering based on statistical characteristics. The atmospheric delay phase (mainly tropospheric delay) caused by the atmospheric phase screen is simulated and removed through meteorological data and GNSS data.

[0128] 2) Deformation signal enhancement. In order to solve the problem of weak landslide hazard signals, the present invention enhances them through two different research perspectives. First, from the perspective of InSAR data processing, phase gradient or (and) trigonometric function decomposition is used to enhance the signal's ability to characterize landslide hazards. Secondly, from the perspective of computer vision, a generative adversarial network is used to perform simulation-level data augmentation for small sample tasks. At the same time, the RGB three-color channel mapping method is also used to reduce the difficulty of network learning.

[0129] 3) Learning deformation signals. In view of the problem that landslide hazard signals are easy to fit but difficult to learn, the present invention strengthens the learning ability of the model by optimizing the network training structure and the reasoning structure. Specifically, in the network training structure, the convolution operation and the self-attention mechanism are integrated to enhance the network's grasp of global information. Secondly, for the reasoning structure, the present invention uses structural reparameterization and selective state space models to optimize the convolution blocks in the training structure to achieve efficient algorithm migration.

[0130] From visual segmentation to semantic understanding. In the identification of landslide hazards, the results identified by the segmentation model (detection model) often need to be further interpreted based on expert knowledge on the landslide type, disaster mechanism and landslide status. There are problems such as high interpretation threshold, reliance on expert knowledge, and lack of self-analysis and self-interpretation mechanism. Therefore, the present invention relates to the collaborative interpretation of different data modalities (images, texts). In order to realize the automated type classification and cause analysis of landslide hazards, the present invention introduces a multimodal image subtitle generation model, combines the InSAR deformation image with the landslide segmentation results, and generates a semantic description by aligning visual and language features. Key technologies include:

[0131] 1) Construct a visual-linguistic representation dataset. Create a dataset of matching landslide images (interference maps and segmentation results) and their corresponding text descriptions to support multimodal learning of image caption generation models. Specifically, first, define description templates and annotate them with geological experts to generate text descriptions. Second, match images and text and perform diversity checks to form visual-linguistic pairs. Finally, augment the visual data and text data separately to prevent potential overfitting risks during training.

[0132] 2) Learn the representation of landslide features. Through multimodal model learning, the visual segmentation results are converted into precise text descriptions to accurately express the characteristics and risks of landslide hazards. Specifically, the landslide feature description is first generated by CLIP (Contrastive Language-Image Pre-training Model) embedding as a prefix. Secondly, a seamless connection from vision to language is constructed through a mapping network. Finally, a landslide prediction description is generated through an automatic regression language generation model.

[0133] From static identification to dynamic interpretation. In order to realize the closed-loop feedback capability of the landslide hazard identification system, the present invention designs an intelligent feedback mechanism based on visual question and answer, which further improves the practicality of the interpretation results through interactive question and answer with users. The key technologies include:

[0134] 1) Construct a visual-language question-answering dataset. Create a dataset that matches landslide images (interference maps and / or segmentation results) and their corresponding interpreted text descriptions, as well as question-answering text descriptions, to support multimodal learning of visual question-answering models. Specifically, first, a description template is defined and annotated by geological experts to generate text descriptions. Second, images and text are matched and diversity checked to form visual-language pairs. Finally, the visual data and text data are augmented separately to prevent potential overfitting risks during training.

[0135] 2) Learn the answers to landslide features. Through multimodal model learning, accurate answer descriptions are generated based on the questions raised by users, and landslide hazards are accurately expressed as readable text. Specifically, first, by encoding data-aware features and language features respectively, formally aligned visual (language) feature vectors are formed. Secondly, through multimodal fusion, the visual (language) feature vectors are weighted and fused with each other. Finally, problem prediction is achieved through a multi-layer perceptron.

[0136] Embodiment 2:

[0137] like Figure 1 As shown, the present invention provides an intelligent identification framework for landslide hazards by combining InSAR technology and deep learning. The complete steps of the method of the present invention include:

[0138] S1: Construction of InSAR semantic segmentation model for entangled phase processing: In view of the potential unwrapping errors and complex operation problems in InSAR data processing, this paper constructs a semantic segmentation model that directly analyzes the interferogram to reduce the dependence of phase unwrapping and parameter tuning on data quality in traditional methods. The specific contents include: 1) Effectively suppress irrelevant phases through geometric modeling, phase filtering and introduction of external data; 2) Effectively highlight the deformation signals of landslide hazards through the perspective of InSAR data processing or (and) computer vision perspective; 3) Strengthen model learning by optimizing the network training structure and reasoning structure; (For details, see Figure 2 )

[0139] S2: Construction of multimodal semantic analysis and image subtitle generation mechanism: In response to the demand for intelligent interpretation of landslide hazards, this invention introduces multimodal image subtitle generation technology, inputs the landslide area segmentation results and InSAR deformation images into the multimodal model, and generates text analysis results by aligning visual and language features to achieve deep semantic analysis of landslide hazards. The specific contents include: 1) Creating a dataset that matches landslide images (interference maps, segmentation results) and their corresponding text descriptions to support multimodal learning of subtitle generation models; 2) Through multimodal model learning, converting visual segmentation results into precise text descriptions to accurately express the characteristics and risks of landslide hazards; (For details, see Figure 3 )

[0140] S3: Construction of intelligent feedback and visual question-answering mechanism: In order to realize the closed-loop feedback capability of the intelligent landslide hazard identification system, the present invention designs an automatic result feedback mechanism after the identification and interpretation stage. By introducing the visual question-answering mechanism, the results of identification and analysis are clearly conveyed to the user in an interactive manner. The specific contents include: 1) Creating a dataset of matching landslide images (interference maps, segmentation results) and their corresponding interpreted text descriptions, as well as question-answering text descriptions, to support multimodal learning of visual question-answering models. 2) Through multimodal model learning, accurate answer descriptions are generated based on the questions raised by the user, accurately expressing the landslide hazards as readable text. (For details, see Figure 4 )

[0141] Construction of InSAR semantic segmentation model for twisted phase processing

[0142] Step S101, suppressing irrelevant phase of the interference pattern.

[0143] This step is aimed at the irrelevant phase interference in the winding phase image. First, the present invention will characterize and remove the interference phase caused by the reference ellipsoid and terrain by means of high-precision geometric modeling. Secondly, for the noise phase, before interference, the phase caused by the Doppler frequency shift is filtered in the azimuth (range) direction, and the speckle noise caused by the coherent superposition of the backscattered waves of multiple scatterers is processed by multi-view processing. After interference, the data is further smoothed by using Goldstein frequency domain filtering based on statistical characteristics. For the atmospheric delay phase (mainly tropospheric delay) caused by the atmospheric phase screen, the present invention simulates and removes it through meteorological data and GNSS data.

[0144] In SAR images, the complex signal of each pixel can be expressed based on the Euler formula: S = A·e jφ Among them, e jφ =cos(φ)+jsin(φ), A represents the amplitude (intensity), represents the phase, R represents the propagation path, λ represents the wavelength of the radar signal, and j is an imaginary unit. For two interferometric SAR images, they can be expressed in complex form:

[0145]

[0146] Among them, S1 represents the first SAR image, and S2 represents the second SAR image. By multiplying the complex conjugate of S1 and S2, we get:

[0147]

[0148] Among them, * represents complex conjugate, A1 represents the amplitude of S1, A2 represents the amplitude of S2, φ1 represents the phase of S1, φ2 represents the phase of S2, and after complex multiplication expansion, the phase difference can be expressed as:

[0149]

[0150] Among them, Δφ represents the phase difference, arg represents the phase angle, and the interference phase can be expressed according to the components as follows:

[0151]

[0152] in, represents the reference ellipsoid phase, represents the terrain phase, represents the deformation phase, represents the atmospheric delay phase, represents the random noise phase.

[0153] Without considering and In case, as well as It can be represented by signal interference modeling. It refers to the interference phase (excluding the flat ground effect) corresponding to the projection point of any point on the ground at equal slant distance on the reference ellipsoid. The observation point P1 is projected onto the reference ellipsoid to P0 with the distance R1 of the main image as the radius and the velocity vector of the satellite flight as the axis. The interference phase at P0 is the interference phase at P1. It can be expressed as:

[0154]

[0155] Where R1 and R2 represent the distances from the first and second imaging points to P1, r1 and r2 represent the distances from the first and second imaging points to P0, B represents the length of the horizontal baseline, B represents the length of the baseline, θ represents the incident angle, and α represents the horizontal angle between the first and second imaging points. It refers to the additional interference phase caused by Δθ caused by the earth height h. The slant range difference ΔR of the two images can be expressed as: ΔR = Bsin(θ-α+Δθ) = Bsin(θ-α)cosΔθ+BsinΔθcos(θ-α). When Δθ→0, we have Therefore, it can be further simplified to: ΔR = Bsin(θ-α) + BΔθcos(θ-α). ΔR is used to represent the interference phase, that is:

[0156]

[0157] Therefore, the terrain phase can be expressed as: Among them, B ⊥ Indicates the length of the vertical baseline. Because the curvature of the earth is small in a small area, Where H is the geodetic height of the first imaging point. After being brought in, the terrain phase can be further simplified as:

[0158]

[0159] SAR satellites are affected by the atmospheric phase screen during signal propagation, which results in an additional propagation path and an atmospheric delay phase. This mainly includes delays caused by the ionosphere and troposphere. The ionosphere delay is caused by changes in electron density and is inversely correlated with the signal wavelength:

[0160]

[0161] in, In this paper, Sentinel-1A / B radar images are mainly used as raw data (L band). Therefore, it is assumed that Right now Dominated by tropospheric delay. The tropospheric delay phase consists of stratified and turbulent components. The stratified component is related to the terrain height and can be removed by considering the terrain factor. The turbulent component is related to the water vapor content and can be removed by introducing meteorological and GNSS data for simulation.

[0162] The interference phase contains a large number of noise phases, which are usually suppressed by filtering. According to the stage of data processing, filtering can be divided into pre-filtering and post-filtering. Pre-filtering is mainly used to suppress noise caused by Doppler decorrelation and speckle effect. Among them, Doppler decorrelation manifests itself as spectral shifts in the azimuth and slant range of the primary and secondary SAR images. This effect can be weakened by pre-filtering in both directions. The speckle effect refers to the speckle of multiplicative noise caused by the coherent superposition of the backscattered waves generated by the scatterers on the incident signal between the scatterers. This noise can be suppressed by multi-look processing:

[0163]

[0164] in, represents the average intensity value after multi-view processing, I represents the intensity value of the i-th view, and L represents the total number of views. Post-filtering is based on the morphological or statistical characteristics of the interference phase to further remove the noise in the interference pattern. The present invention will use Goldstein frequency domain filtering to further remove the noise in the interference pattern:

[0165] H(u,v)=S{Z(u,v)} α Z(u,v) (10)

[0166] where a∈[0,1] represents the filter parameter; S{·} is the smoothing operator; u and v are the spatial frequencies; Z(u,v) and H(u,v) are the Fourier spectra of the interference pattern before and after filtering, respectively.

[0167] The interference results can be converted into The interference phase is dominant. Perform relevant enhancement operations. First, enhance the sensitivity of the data to deformation perception by means of phase gradient. Phase gradient refers to the gradient (i.e., rate of change) of the phase difference between adjacent pixels in the interference pattern, which can be used to represent the intensity and direction of local surface deformation:

[0168]

[0169] in, Indicates the phase value along the direction x The gradient of represents the gradient of the phase value along the direction y, φ(x,y) represents the phase value of a pixel position (x,y) in the interference pattern, φ(x+1,y)-φ(x,y) and φ(x,y+1)-φ(x,y) represent the phase value in x , the phase difference between adjacent pixels in the y direction, Δx and Δy represent the phase difference along the direction x , pixel spacing in direction y;

[0170] Step S102: enhancing the deformation signal.

[0171] This step is aimed at the problem of weak signal of landslide hazard. The present invention enhances it through two different research perspectives. First, from the perspective of InSAR data processing, the phase gradient or (and) trigonometric function decomposition can be used to enhance the signal's ability to characterize landslide hazards. Second, from the perspective of computer vision, a generative adversarial network can be used to perform simulation-level data augmentation on small sample tasks. At the same time, the RGB three-color channel mapping method can also be used to reduce the difficulty of network learning.

[0172] First, the ability to express deformation features of the data is enhanced by color rendering. In a single-channel interference image, the information dimension is limited and cannot fully express the complex spatial changes that are easily recognized by the human eye. After mapping the single-channel phase or phase gradient data to an RGB image, different color channels can be used to encode information such as the strength and direction of the phase gradient, thereby enhancing the ability to express features. Specifically, first standardize the interference phase to between [0,1]:

[0173]

[0174] Among them, φ and φ norm Respectively represent the phase before and after normalization. Then, a discrete color list C = {c i} i＝1,2,...,n , where each c i =(R i ,G i ,B i ), and calculate the RGB value by linear interpolation. In the semantic segmentation task, the convolution operation in CNNs extracts the local features of the image through the convolution kernel. In the convolution operation, the input data (U and V represent the length and width of the input, respectively, and C represents the number of input channels) and the convolution kernel (H and W represent the length and width of the convolution kernel, respectively, and D represents the number of output channels) and add a bias matrix (R and T represent the length and width of the output, B z represents the bias of the z-th convolution) to generate the output The value of its z-th channel can be expressed as:

[0175]

[0176] Among them, O (：,：,z) Represents the output value of the zth channel, M (：,：,k) represents the input value of the kth channel, * represents the convolution operation, F (：,：,k,z) Indicates that (：,：,z) and M (：,：,k) The corresponding convolution kernel, B (：,：,z) is a biased z The matrix formed.

[0177] However, a fixed convolution kernel size will limit the convolution to a fixed receptive field and fail to capture global features. The self-attention mechanism integrates long-distance dependency information into the model by calculating the correlation between different positions in the feature map, enabling it to effectively integrate features in spatial and temporal dimensions. In the self-attention mechanism, the input sequence is first transformed into three matrices through three linear transformations: the query matrix Q, the key matrix K, and the value matrix V. Then, the attention matrix (d k is the dimension of Q and K):

[0178]

[0179] Where Attention(Q,K,V) represents the value of the self-attention mechanism, d k is the dimension of Q and K, and softmax is the activation function;

[0180] Step S103: learning deformation signals.

[0181] This step is aimed at the problem that landslide hazard signals are easy to fit but difficult to learn. The present invention strengthens the learning ability of the model by optimizing the network training structure and the reasoning structure. Specifically, in the network training structure, the fusion of convolution operations and self-attention mechanisms is considered to enhance the network's grasp of global information. Secondly, for the reasoning structure, the present invention uses structural reparameterization and selective state space models to optimize the convolution blocks in the training structure to achieve algorithm migration. The present invention deeply integrates CNNs with the self-attention mechanism, combines spatial features and global dependencies, and enhances the detection ability of landslides in complex terrain areas. In specific implementation, a multi-level feature extraction module combining CNNs and self-attention mechanisms is designed. The local boundary features of the landslide area are captured by convolution operations, and the information flow of cross-regional features is realized with the help of the self-attention mechanism. This module extracts and fuses the spatial structural features and deformation information of the landslide area layer by layer by stacking multiple layers of deep networks.

[0182] According to the convolution calculation formula in the previous article, it can be inferred that convolution only involves linear operations in the entire calculation process, so it is a linear operation. For convolutions with the same structure, they are distributive and additive:

[0183] Let F'←cF, then c(M*F)=M*F' (15)

[0184] M*F 1 +M*F 2 =M*(F 1 +F 2 ), let F'←F 1 +F 2 , there is M*F 1 +M*F 2 =M*F' (16)

[0186] Where M represents an arbitrary constant, F, F 1 、F 2 Represents different convolution kernels, and * represents convolution operations. Structural reparameterization is an optimization technique based on the distribution and additivity of homogeneous convolution operations. Its core lies in adopting complex structures in the training phase to enhance feature learning and model expression capabilities, and simplifying them into efficient equivalent structures in the inference phase, thereby reducing computational overhead and improving inference efficiency. This technology is mainly applied to the convolution operation part in the multi-level feature extraction module, so that the segmentation network can contain multi-branch, redundant convolution kernels or non-standard structures during training to better adapt to the data and enhance the robustness of the model. In the inference phase, these redundant structures are merged into a simpler equivalent structure to accelerate inference while maintaining performance during training.

[0187] A state space is the minimum number of variables that contain a complete description of a system. It is a way to represent a problem mathematically by defining the possible states of a system. State space models are used to describe these states and predict the next state of the system based on the inputs:

[0188]

[0189] Among them, A, B, C, and D all represent coefficient matrices, h(t) and h'(t) represent the current and next states respectively, x(t) represents the input, and y(t) represents the output. For matrix A, in order to keep a large historical memory, the high-order polynomial projection HiPPO operator is used to construct it:

[0190]

[0191] Among them, n and k are used to define the index of A. In order for SSM to receive (discrete) sequence data and generate sequence output, it is necessary to replace the matrix of the computation state with a discrete matrix:

[0192]

[0193] Where Δ represents the discretized time step, is the discretized matrix, and I represents the unit matrix. At the same time, in order to selectively compress the data into the state, even if the matrices B, C, and Δ depend on the input, the selective SSM divides the entire time series into multiple shorter blocks, each containing multiple consecutive time steps. Within each block, the state update still maintains the recursive order to ensure the temporal dependency of the data. In the segmentation model, combining the selective SSM with the self-attention mechanism can avoid the computational bottleneck of self-attention in long sequence processing, so that the segmentation model is improved in both efficiency and accuracy.

[0194] Constructing a multimodal semantic parsing and image caption generation mechanism

[0195] Step S201: Constructing a visual-language representation dataset

[0196] Define a set of templates τ = [T1, T2, ..., T n ] Each template T i Contains variable placeholders for inserting the characteristic information of the landslide. The template format is as follows:

[0197] T i ='Landslide type <category>, landslide characteristics <evaluation>'(20)

[0198] Among them, <type> is used to record the type of landslide, and <evaluation> is used to record the characteristic information such as the disaster mechanism and state of the landslide. According to the interference map and segmentation results, the image features F = [f1, f2, ..., f n ], and the template is filled in by geological experts based on the features. Subsequently, image-text matching is performed, and the image features F and the text template τ are matched to generate a paired data set P = {(f1, T1), (f2, T2), ..., (f n ,T n )}. Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure. Finally, the dataset is augmented: for visual data, geometric transformation, color transformation and noise injection are used for augmentation; for text data, synonym replacement, variant description and other methods are used for augmentation. At this point, the final dataset is output. in, and Represent the augmented visual features and text descriptions respectively.

[0199] Step S202: learning landslide characteristic description.

[0200] This step is based on a multimodal learning framework and uses a visual encoder, a language encoder, and an attention mechanism to build a multimodal caption generation model. The visual encoder encodes the input landslide hazard segmentation results and their corresponding interference phases into a visual feature vector V. The language encoder maps the description text of the landslide hazard area to the same semantic space and encodes it into a language feature vector L:

[0201]

[0202] Among them, x i ,y i represents the value of the index at the ith position of the (entangled) InSAR interferogram and the landslide potential area segmentation result produced by S1, z i Indicates the value of the index at the i-th position of the landslide text description. vision and Encoder language denote the visual and language encoders respectively.

[0203] This step maps V and L to the same embedding space by constructing a shared feature space. The training goal is to maximize the cosine similarity of similar image and text pairs while minimizing the similarity of non-matching pairs:

[0204]

[0205] Among them, ||·|| represents the Euclidean norm, and Sim(V,L) represents the similarity between visual features and language features. Afterwards, a lightweight mapping network is used to map the embedding vector of the feature space (from CLIP) into k embedding vectors:

[0206]

[0207] Among them, F(CLIP(x i )) means input x i The feature vectors generated by the CLIP model are further mapped to the function of the embedding space. Each vector Has the same dimensions as the word embeddings. And is concatenated with the visual embeddings to the caption embeddings:

[0208]

[0209] This step is done by concatenating the prefix and the caption sequence during training. Input into the language model. The model is made to predict the token of the title in an autoregressive manner, conditioned on the prefix. To this end, this step uses a simple but effective cross entropy loss to train the mapping component.

[0210] Building intelligent feedback and visual question-answering mechanisms

[0211] Step S301: Constructing a visual-language question answering dataset

[0212] The question-answer text description includes questions Q and answers A. First, define the question template set T Q =[T Q1 ,T Q2 ,...,T Qn ], where each question template T Qi Questions are asked about the specific characteristics of the landslide. Then, a set of answer templates T is defined A =[T A1 ,T A2 ,...,T An ], where each question template T Ai The corresponding features extracted from the landslide image are filled in, and the question and answer form a question-answer pair (Q, A) = (T Q (F),T A (F)). Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure. Finally, the dataset is augmented. At this point, the final dataset is output. in, and Represent the augmented visual features and text (questions and corresponding answers) respectively.

[0213] Step S302: Learning landslide characteristics Answer:

[0214] In this step, a landslide hazard VQA model is designed, which includes three steps: feature extraction, fusion, and prediction. In the feature extraction part, ViT and BERT models are mainly used to extract the features of images and texts respectively. The image input is divided into fixed-size tiles by ViT and encoded into a series of embedded representations, defined as:

[0215]

[0216] Among them, d v represents the embedding dimension, and K represents the number of tokens after segmentation. For a given question text input, the BERT model segments the text into several word embeddings, defined as:

[0217]

[0218] Among them, dq represents the dimension of embedding, and M represents the number of tokens after chunking. Subsequently, the context-dependent embedding is generated by introducing self-attention and cross-attention mechanisms. Subsequently, the text representation Q generated by BERT can be given i The attention weights are calculated through two fully connected layers and the activation function ReLU:

[0219]

[0220] in, represents the attention weight score of the mth token, and are the learned weight matrices respectively, and then, the attention weights are used to represent the text q m Perform weighted summation to get attention-weighted text embedding:

[0221]

[0222] in, Represents the attention weight score of the mth token after softmax activation. Then, for the image embedding V generated by ViT i , first combining it with the attention-weighted text representation q:

[0223] v' k =(W v v k )⊙(W q q) (29)

[0224] in, and is the linear transformation matrix. Then the combined image can be represented by v′ k Calculate attention weights.

[0225] This step performs a weighted summation of the image embeddings to obtain the attention-weighted image embedding:

[0226]

[0227] in, represents the k-th image embedding v k Attention weights. In the fusion stage, you can choose MUTAN (Multimodal Tucker Fusion) to combine image and text embeddings. MUTAN uses Tucker decomposition to fuse the tensor Decomposed into three matrices and a small kernel tensor:

[0228] T=((T c ×1 W q )×2 W v )×3 Wo (31)

[0229] in, They are matrices for preparing the fusion process. The design of MUTAN can effectively capture the subtle differences in information in the combination of visual and language data, enabling the model to better handle remote sensing question answering tasks.

[0230] This step passes the fused embedding into the multi-layer perceptron, with each dimension corresponding to a possible answer category. The model also integrates multi-task learning at this stage, enhancing the adaptability of the model by training the attention module and classification weights for each question type separately:

[0231]

[0232] Among them, y represents the final output, MLP represents multi-layer perceptron, Represents a multimodal fusion mechanism for integrating image features And the text feature q is fused.

[0233] Through the above steps, the method of the present invention improves the question-answering performance of the model in the RSVQA task while maintaining the combination of multimodal data. The combination of these techniques and formulas enables the model to dynamically understand the relationship between visual data and text queries, thereby providing accurate answers to complex remote sensing visual question-answering.

[0234] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0235] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. An intelligent identification method for creep landslide hazards combining image processing and semantic understanding, characterized in that: The method comprises: S1: Input the winding InSAR interferogram; first, through geometric modeling, phase filtering and the introduction of external data, the image is processed to suppress the irrelevant phase caused by the earth's ellipsoid, terrain, atmosphere and Doppler frequency shift; then, the deformation signal of landslide hazard is highlighted by calculating the deformation phase gradient, signal trigonometric function decomposition and RGB three-color channel mapping; finally, the deep learning semantic segmentation method of joint convolution operation, self-attention mechanism, structural reparameterization and selective state space model technology is used to realize the identification of landslide hazards in the target area; output the raster and vector data of the landslide hazard area identified by the deep learning semantic segmentation method; S2: Input the winding InSAR interferogram and the landslide hazard area segmentation result produced by step S1; first, generate a text description of the landslide hazard area by defining a template and expert annotation, including landslide type, landslide disaster mechanism, and landslide state characteristics; then, generate a feature description of the landslide hazard by a deep learning image subtitle generation method combining vision and text processing; output the text description of the landslide hazard area interpreted by the deep learning image subtitle generation method; S3: Input the landslide hazard area segmentation result produced by step S1 and the text description produced by step S2; first, generate a landslide hazard area question-answer dataset by defining templates and expert annotations; finally, generate an interactive question-answer mechanism for the landslide hazard area by a deep learning visual question-answering mechanism method that combines vision and text processing; output the interpretation and judgment of the landslide hazard area and related issues made by the deep learning visual question-answering mechanism method.

2. According to the method for intelligent identification of creep landslide hazards combining image processing and semantic understanding according to claim 1, it is characterized in that: The step S1 comprises: S101: Through high-precision geometric modeling, filtering and external data correction, irrelevant phase interference in the interference pattern is suppressed, the pure interference phase information related to landslide hazards is retained, and the filtered and corrected interference pattern is generated to provide high-quality input data for subsequent deformation signal enhancement and learning; S102: Generate RGB images or other high-dimensional feature data containing enhanced landslide hazard deformation features from InSAR data through phase gradient enhancement, RGB channel mapping and generative adversarial network methods, providing easy-to-learn and identifiable landslide deformation signal representation for semantic segmentation networks; S103: By integrating convolution operations, self-attention mechanism and structural reparameterization methods, the training and reasoning structures are optimized. The optimized deep learning model has a segmentation network with accurate identification of landslide hazard areas and efficient reasoning capabilities, and outputs accurate segmentation results of landslide hazards.

3. According to claim 1, a method for intelligent identification of creep landslide hazards combining image processing and semantic understanding is characterized in that: The step S2 comprises: S201: Create a dataset of matching landslide images, i.e., interference maps and segmentation results, and their corresponding text descriptions to support multimodal learning of the IC model; first, by defining description templates and annotating them by geological experts to generate text descriptions, second, matching images and texts and performing diversity checks to form visual-language pairs, and finally, by augmenting the visual data and text data separately to prevent potential overfitting risks during training; S202: By constructing a multimodal subtitle generation model, the landslide hazard segmentation results and interference phases are encoded into visual feature vectors using a visual encoder, and the description text is mapped into a language feature vector using a language encoder, thereby generating a landslide feature description, including visual feature vectors from the landslide hazard segmentation results and interference phases, and language feature vectors from the description text, providing basic data for subsequent feature alignment and language generation; by constructing a shared feature space, the visual feature vectors and language feature vectors are mapped to the same embedding space to maximize the similarity of matching image and text pairs, and a lightweight mapping network is combined to generate an embedding vector to form a seamless connection from vision to language; by inputting the connection sequence into the autoregressive language generation model, the landslide description is predicted under the prefix condition, so that the cross entropy loss function is used to train the generation component to optimize the prediction effect, and finally an accurate text description of the landslide hazard is output.

4. The method for intelligent identification of creep landslide hazards by combining image processing and semantic understanding according to claim 1 is characterized in that: The step S3 comprises: S301: Create a dataset of matching landslide images and their corresponding interpretation text descriptions, as well as question-answering text descriptions, to support multimodal learning of the VQA model. Specifically, first, define description templates and generate text descriptions through annotation by geological experts. Second, match images and texts and perform diversity checks to form visual-language pairs. Finally, augment visual data and text data separately to prevent potential overfitting risks during training. S302: First, use the ViT and BERT models to encode the features of image and text data respectively, generate form-aligned visual feature vectors and language feature vectors, and form a preliminary embedding representation; second, use the multimodal fusion method to make the visual feature vector and the language feature vector mutually weighted and fused to capture subtle differences in information; finally, input the fused embedding into the multi-layer perceptron to achieve accurate prediction of the problem, and enhance the model's adaptability to different types of problems through multi-task learning.

5. The method for intelligently identifying creep-type landslide hazards by combining image processing and semantic understanding according to claim 2 is characterized in that: The step S1 specifically includes: S101: Suppressing irrelevant phase of the interferogram: For irrelevant phase interference in the winding phase image; first, the interference phase caused by the reference ellipsoid and terrain is characterized and removed by high-precision geometric modeling; second, for the noise phase, before interference, the phase caused by Doppler frequency shift is filtered in azimuth, and the speckle noise caused by the coherent superposition of backscattered waves from multiple scatterers is processed by multi-look processing. After interference, the data is further smoothed by Goldstein frequency domain filtering based on statistical characteristics. For the atmospheric delay phase caused by the atmospheric phase screen, it is simulated and removed by meteorological data and GNSS data; In SAR images, the complex signal of each pixel can be expressed based on the Euler formula: S = A·e jφ , where e jφ =cos(φ)+jsin(φ), A represents the amplitude, represents the phase, R represents the propagation path, λ represents the wavelength of the radar signal, j is the imaginary unit, and for two interferometric SAR images, they are expressed in complex form: Among them, S1 represents the first SAR image, S2 represents the second SAR image, and by multiplying S1 and S2 by complex conjugate, we get: Among them, * represents complex conjugate, A1 represents the amplitude of S1, A2 represents the amplitude of S2, φ1 represents the phase of S1, φ2 represents the phase of S2, and after complex multiplication expansion, the phase difference can be expressed as: Among them, Δφ represents the phase difference, arg represents the phase angle, and the interference phase can be expressed according to the components as follows: in, represents the reference ellipsoid phase, represents the terrain phase, represents the deformation phase, represents the atmospheric delay phase, represents the phase of random noise; Without considering and In case, as well as It can be represented by signal interference modeling. It refers to the interference phase corresponding to the equidistant projection point of any point on the ground on the reference ellipsoid. The observation point P1 is projected onto the reference ellipsoid to P0 with the distance R1 of the main image as the radius and the velocity vector of the satellite flight as the axis. The interference phase at P0 is the interference phase at P1. It can be expressed as: Among them, R1 and R2 represent the distances from the first and second imaging points to P1, r1 and r2 represent the distances from the first and second imaging points to P0, B represents the length of the horizontal baseline, B represents the length of the baseline, θ represents the incident angle, and α represents the horizontal angle between the first and second imaging points. It refers to the additional interference phase caused by Δθ caused by the earth height h. The slant range difference ΔR of the two images can be expressed as: ΔR = Bsin(θ-α+Δθ) = Bsin(θ-α)cosΔθ+BsinΔθcos(θ-α). When Δθ→0, we have Therefore, it can be further simplified to: ΔR = Bsin(θ-α) + BΔθcos(θ-α), and ΔR is used to represent the interference phase, that is: Therefore, the terrain phase can be expressed as: Among them, B ⊥ Indicates the length of the vertical baseline. Because the curvature of the earth is small in a small area, Among them, H represents the geodetic height of the first imaging point, so After being brought in, the terrain phase can be further simplified as: SAR satellites are affected by the atmospheric phase screen during signal propagation, resulting in additional propagation paths, which in turn produce atmospheric delay phases, mainly including delays caused by the ionosphere and troposphere. The ionosphere delay is caused by changes in electron density and is inversely correlated with the signal wavelength: in, represents the ionospheric delayed phase, TEC represents the total electron content, and the Sentinel-1A / B radar image is used as the original data, that is, the L band. Therefore, it is assumed that Right now The tropospheric delay is dominated by the tropospheric delay phase, which consists of stratified and turbulent components. The stratified component is related to the terrain height and is removed by considering the terrain factor. The turbulent component is related to the water vapor content and is removed by introducing meteorological and GNSS data for simulation. Interference phase filtering can be divided into pre-filtering and post-filtering according to the stage of data processing; pre-filtering is used to suppress the noise caused by Doppler decorrelation and speckle effect; among them, Doppler decorrelation is manifested as spectrum shift of primary and secondary SAR images in azimuth and slant range, and this effect is weakened by pre-filtering in two directions respectively. The speckle effect refers to the speckle of multiplicative noise caused by the coherent superposition of the backscattered waves generated by the scatterers on the incident signal between the scatterers. This noise can be suppressed by multi-look processing: in, represents the average intensity value after multi-view processing, I represents the intensity value of the i-th view, L represents the total number of views, and post-filtering is based on the morphological or statistical characteristics of the interference phase to further remove the noise in the interference pattern; Goldstein frequency domain filtering will be used to further remove the noise in the interference pattern: H(u,v)=S{|Z(u,v)|} α Z(u,v) (10) Where a∈[0,1] represents the filter parameter; S{·} is the smoothing operator; u and v are the spatial frequencies; Z(u,v) and H(u,v) are the Fourier spectra of the interference pattern before and after filtering, respectively; The interference results are converted into The interference phase is dominant. Perform related enhancement operations; first, enhance the sensitivity of the data to deformation perception by means of phase gradient. Phase gradient refers to the gradient of the phase difference between adjacent pixels in the interference pattern, that is, the rate of change, which indicates the intensity and direction of local surface deformation: in, Indicates the phase value along the direction x The gradient of represents the gradient of the phase value along the direction y, φ(x,y) represents the phase value of a pixel position (x,y) in the interference pattern, φ(x+1,y)-φ(x,y) and φ(x,y+1)-φ(x,y) represent the phase differences between adjacent pixels in the x and y directions, respectively, and Δx and Δy represent the pixel spacing along the direction x and direction y, respectively; S102: Enhanced deformation signal: In order to solve the problem of weak signal of landslide hazard, two different research perspectives are used to enhance it. First, from the perspective of InSAR data processing, phase gradient and trigonometric function decomposition are used to enhance the signal's ability to represent landslide hazards. Second, from the perspective of computer vision, generative adversarial networks are used to perform data augmentation at the simulation level for small sample tasks. At the same time, RGB three-color channel mapping can also be used to reduce the difficulty of network learning. First, the ability of the data to express deformation features is enhanced by color rendering. After mapping the single-channel phase or phase gradient data to an RGB image, different color channels are used to encode the strength and direction information of the phase gradient, thereby enhancing the feature expression capability. Specifically, the interference phase is first standardized to between [0,1]: Among them, φ and φ norm Represent the phase before and after normalization respectively. Then, a discrete color list C = {c i } i＝1,2,...,n , where each c i =(R i ,G i ,B i ), and calculate the RGB value by linear interpolation; in the semantic segmentation task, the convolution operation in CNNs extracts the local features of the image through the convolution kernel; in the convolution operation, the input data With convolution kernel Convolution is performed, U and V represent the length and width of the input, C represents the number of input channels, H and W represent the length and width of the convolution kernel, D represents the number of output channels, and a bias matrix is ​​added R and T represent the length and width of the output, B z Represents the bias of the zth convolution to generate the output The value of its z-th channel can be expressed as: Among them, O (：,：,z) Represents the output value of the zth channel, M (：,：,k) represents the input value of the kth channel, * represents the convolution operation, F (：,：,k,z) Indicates that (：,：,z) and M (：,：,k) The corresponding convolution kernel, B (：,：,z) is a biased z The matrix formed is: In the self-attention mechanism, the input sequence first undergoes three linear transformations to obtain three matrices, the query matrix Q, the key matrix K, and the value matrix V, and then calculates the attention: Where Attention(Q,K,V) represents the value of the self-attention mechanism, d k is the dimension of Q and K, and softmax is the activation function; Step S103: Learning deformation signal: The learning ability of the model is enhanced by optimizing the network training structure and the inference structure. Specifically, in the network training structure, the convolution operation and the self-attention mechanism are integrated to enhance the network's grasp of global information. Secondly, for the inference structure, the convolution blocks in the training structure are optimized using structural reparameterization and selective state space models to achieve algorithm transfer. CNNs are deeply integrated with the self-attention mechanism, and the spatial features and global dependencies are combined to enhance the detection ability of landslides in complex terrain areas. In terms of specific implementation, a multi-level feature extraction module combining CNNs and the self-attention mechanism is designed to capture the local boundary features of the landslide area through convolution operations, and then the information flow of cross-regional features is realized with the help of the self-attention mechanism. This module extracts and fuses the spatial structure features and deformation information of the landslide area layer by layer through the stacking of multiple layers of deep networks. According to the above convolution calculation formula, convolution only involves linear operations in the entire calculation process, so it is a linear operation. For convolutions with the same structure, they are distributive and additive: M*F 1 +M*F 2 =M*(F 1 +F 2 ), let F'←F 1 +F 2 , there is M*F 1 +M*F 2 =M*F'(16) Where M represents an arbitrary constant, F, F 1 、F 2 Represents different convolution kernels, and * represents convolution operation. Structural reparameterization is applied to the convolution operation part in the multi-level feature extraction module, so that the segmentation network can contain multiple branches, redundant convolution kernels or non-standard structures during training. In the inference stage, these redundant structures are merged into a simpler equivalent structure, thereby accelerating inference while maintaining the performance during training. A state space is the minimum number of variables that contain a complete description of a system; it is a way to represent a problem mathematically by defining the possible states of the system; a state space model is used to describe these states and predict the next state of the system based on the inputs: Among them, A, B, C, and D all represent coefficient matrices, h(t) and h'(t) represent the current and next states respectively, x(t) represents the input, and y(t) represents the output. For matrix A, in order to keep a large historical memory, the high-order polynomial projection HiPPO operator is used to construct it: Among them, n and k are used to define the index of A; in order for SSM to receive sequence data and generate sequence output, it is necessary to replace the matrix of the calculation state with a discrete matrix: Where Δ represents the discretized time step, is the discretized matrix, I represents the unit matrix; at the same time, the data is selectively compressed into the state, even if the matrices B, C and Δ depend on the input, the selective SSM divides the entire time series into multiple shorter blocks, each block contains multiple consecutive time steps; within each block, the state update still maintains the recursive order to ensure the temporal dependency of the data; in the segmentation model, the selective SSM is combined with the self-attention mechanism.

6. The method for intelligent identification of creep landslide hazards by combining image processing and semantic understanding according to claim 3 is characterized in that: The step S2 specifically includes: Step S201: Constructing a visual-language representation dataset: Define a set of templates τ = [T1, T2, ..., T n ] Each template T i It includes variable placeholders for inserting the characteristic information of the landslide. The template format is as follows: T i ='Landslide type <category>, landslide characteristics <evaluation>'(20) Among them, <type> is used to record the type of landslide, and <evaluation> is used to record the disaster mechanism and state characteristic information of the landslide; according to the interference map and segmentation results, the image features F = [f1, f2, ..., f n ], and the geological experts fill in the templates with features; then, image-text matching is performed, and the image features F and the text template τ are matched to generate a paired data set P = {(f1, T1), (f2, T2), ..., (f n ,T n )}; Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure; Finally, the dataset is augmented: for visual data, geometric transformation, color transformation and noise injection are used for augmentation; for text data, synonym replacement and variant description are used for augmentation; At this point, the final dataset is output in, and Represent the augmented visual features and text descriptions respectively; Step S202: Learning landslide feature description: Based on the multimodal learning framework, a multimodal caption generation model is constructed using a visual encoder, a language encoder and an attention mechanism. The visual encoder encodes the input landslide hazard segmentation result and its corresponding interference phase into a visual feature vector V. The language encoder maps the description text of the landslide hazard area to the same semantic space and encodes it into a language feature vector L: Among them, x i ,y i represents the value of the index at the ith position of the (entangled) InSAR interferogram and the landslide potential area segmentation result produced by S1, z i Indicates the value of the index at the i-th position of the landslide text description; Encoder vision and Encoder language denote the visual and language encoders respectively; By constructing a shared feature space, V and L are mapped to the same embedding space; the training goal is to maximize the cosine similarity of similar image and text pairs, while minimizing the similarity of non-matching pairs: Among them, ||·|| represents the Euclidean norm, Sim(V,L) represents the similarity between visual features and language features, and then a lightweight mapping network is used to map the embedding vector of the feature space into k embedding vectors: Among them, F(CLIP(x i )) means input x i The feature vectors generated by the CLIP model are further mapped to the function of the embedding space; each vector Has the same dimensions as word embeddings; and is concatenated with the visual embeddings: By concatenating the prefix with the caption sequence during training Input into the language model, so that the model predicts the token of the title in an autoregressive manner under the condition of the prefix; use cross entropy loss to train the mapping component.

7. The method for intelligently identifying creep-type landslide hazards by combining image processing and semantic understanding according to claim 4 is characterized in that: The step S3 specifically includes: Step S301: Constructing a visual-language question answering dataset: The question-answer text description includes questions Q and answers A. First, define the question template set T Q =[T Q1 ,T Q2 ,...,T Qn ], where each question template T Qi Questions are asked about the specific characteristics of the landslide; then, a set of answer templates T is defined A =[T A1 ,T A2 ,...,T An ], where each question template T Ai The corresponding features extracted from the landslide image are filled in, and the question and answer form a question-answer pair (Q, A) = (T Q (F),T A (F)); Then, the dataset is checked for diversity, including visual diversity based on image feature distribution and language diversity based on syntactic structure; Finally, the dataset is augmented; At this point, the final dataset is output in, and Represent the augmented visual features and text (questions and corresponding answers) respectively; Step S302: Learning landslide characteristics Answer: A landslide hazard VQA model is designed, which includes three steps: feature extraction, fusion and prediction. In the feature extraction part, ViT and BERT models are used to extract the features of images and texts respectively. The image input is divided into fixed-size tiles by ViT and encoded into a series of embedded representations, which are defined as: Among them, d v represents the dimension of embedding, and K represents the number of tokens after segmentation. For a given question text input, the BERT model divides the text into several word embeddings, defined as: Among them, d q represents the dimension of embedding, M represents the number of tokens after segmentation; self-attention and cross-attention mechanisms are introduced to generate context-dependent embeddings; given the text representation Q generated by BERT i The attention weights are calculated through two fully connected layers and the activation function ReLU: in, represents the attention weight score of the mth token, and are the learned weight matrices respectively, and then, the attention weights are used to represent the text q m Perform weighted summation to get attention-weighted text embedding: in, Represents the attention weight score of the mth token activated by softmax, and then, for the image embedding V generated by ViT i , first combining it with the attention-weighted text representation q: v' k =(W v v k )⊙(W q q) (29) in, and is the linear transformation matrix; then the combined image is represented by v′ k Calculate attention weights; The image embeddings are weighted summed to obtain the attention-weighted image embedding: in, represents the k-th image embedding v k Attention weights, in the fusion stage, MUTAN is selected to combine image and text embeddings. MUTAN uses Tucker decomposition to fuse the tensor Decomposed into three matrices and a small kernel tensor: in, They are the matrices for preparing the fusion process; The fused embedding is passed into a multi-layer perceptron, with each dimension corresponding to a possible answer category. The model also integrates multi-task learning at this stage to enhance the adaptability of the model by training the attention module and classification weights for each question type separately: Among them, y represents the final output, MLP represents multi-layer perceptron, Represents a multimodal fusion mechanism for integrating image features And the text feature q is fused.

Images