A method for constructing a multi-modal landslide space-time prediction model

Abstract

A kind of construction method of multi-modal landslide space-time prediction model, including preparing multi-modal landslide data set;Multi-modal landslide data set is preprocessed, and multi-modal landslide sample set is generated;AnyGPT framework is used as basic model, and the basic model includes multi-modal encoder, multi-modal language model and multi-modal decoder;Multi-modal encoder includes 3D convolutional neural network encoder, long short-term memory network encoder and Transformer encoder;3D convolutional neural network encoder is replaced by MobileNet encoder, and Transformer encoder is replaced by sparse Transformer encoder, GRPO reinforcement learning prediction decision is set between multi-modal language model and multi-modal decoder, and improved model is constructed;Multi-modal landslide sample set is classified into training set, verification set and test set, and the improved model is trained and verified, and multi-modal landslide space-time prediction model based on AnyGPT-SLIDE is generated.The combination of setting GRPO reinforcement learning and AnyGPT framework is realized by the present application, and dynamic adjustment prediction strategy is realized.

Description

Technical Field

[0001] This invention relates to the field of geological disaster prediction technology, specifically to a method for constructing a multimodal landslide spatiotemporal prediction model. Background Technology

[0002] In the field of landslide disaster prediction, the fusion and dynamic evolution analysis of multi-source data has always been a core research direction. Currently, multimodal information acquired by InSAR technology, including surface deformation rate data, meteorological factors (rainfall, temperature, etc.), geological data (slope, vegetation cover, etc.), and historical landslide records, has become a key basis for analyzing the causes and mechanisms of landslides. However, existing technologies still have significant limitations in processing this data: on the one hand, different data sources exhibit significant spatial heterogeneity and complex temporal dependencies. Traditional feature methods, limited by design rules, struggle to overcome the heterogeneity barrier of cross-modal data and cannot efficiently model the spatiotemporal coupling patterns between surface deformation and factors such as meteorology and topography. On the other hand, supervised learning models based on static labels show insufficient adaptability in dynamic geological environments and struggle to cope with the nonlinear dynamic characteristics in the landslide evolution process. Especially in multi-stage decision-making scenarios, the strategy optimization capability is limited, resulting in prediction accuracy and generalization that fail to meet practical needs.

[0003] Against this backdrop, multimodal processing frameworks based on the "any-to-any" concept have gradually gained attention. Their core lies in achieving unified representation and interactive understanding of data sources of different types and dimensions, with the AnyGPT framework being a typical embodiment of this idea. By constructing a universal encoding and decoding mechanism, heterogeneous data is transformed into an interactive, unified feature space, thereby breaking down information barriers between modalities. However, in existing landslide prediction technologies, the application of such frameworks remains largely unexplored: traditional models lack efficient discretization capabilities for high-dimensional continuous features (such as terrain parameters and InSAR deformation rates) and have failed to establish dynamic correlation learning mechanisms between multimodal data. This prevents the cross-modal information fusion advantages under the "any-to-any" concept from being fully realized, significantly limiting the adaptability and prediction accuracy of landslide prediction models to complex geological environments.

[0004] Therefore, it is necessary to design a technical solution for landslide risk prediction based on the AnyGPT framework to realize the "any-to-any" cross-modal interaction concept. Summary of the Invention

[0005] To address the shortcomings of the existing AnyGPT framework in landslide prediction technology, including its inability to leverage the advantages of "any-to-any" cross-modal information fusion, which severely limits the adaptability and prediction accuracy of current landslide prediction models, this invention provides a method for constructing a multimodal landslide spatiotemporal prediction model, comprising: Prepare a multimodal landslide dataset; the multimodal landslide dataset includes InSAR deformation data, meteorological data, topographic parameters, and historical landslide records; The multimodal landslide dataset is preprocessed to generate a multimodal landslide sample set; The AnyGPT framework is used as the base model, which includes a multimodal encoder, a multimodal language model, and a multimodal decoder. The multimodal encoder maps data from different modalities to a common feature space and includes a 3D convolutional neural network encoder, a long short-term memory network encoder, and a Transformer encoder. The multimodal language model enables cross-modal fusion and outputs fused features containing spatiotemporal coupling information. The multimodal decoder outputs the final landslide probability prediction result. The 3D convolutional neural network encoder in the base model is replaced with a MobileNet encoder, the Transformer encoder is replaced with a sparse Transformer encoder, and GRPO reinforcement learning prediction decision is set between the multimodal language model and the multimodal decoder to construct an improved model, which is an AnyGPT-SLIDE network structure. The multimodal landslide sample set is classified into a training set, a validation set, and a test set. The improved model is then trained and validated to generate a multimodal landslide spatiotemporal prediction model based on AnyGPT-SLIDE. The MobileNet encoder is an InSAR branch used to process slope direction deformation rate sequences; the Long Short-Term Memory Network encoder is a meteorological branch used to process meteorological time-series data; and the Sparse Transformer encoder is a terrain branch used to process discrete token sequences of terrain parameters.

[0006] Furthermore, the preprocessing includes the following: For the InSAR deformation data, the slope direction deformation rate sequence is obtained using SBAS-InSAR technology; For the meteorological data, the daily time-series data in the meteorological data is converted into time-series data at a specified scale by means of sliding window aggregation, and the specified scale includes weekly scale or monthly scale; For terrain parameters, a discrete token sequence is obtained by introducing a vector quantization variational autoencoder (VQ-VAE). Historical landslide records are tagged, with the tags including the landslide location from the previous year and the activity intensity information for the previous period.

[0007] Furthermore, the process of obtaining the discrete token sequence includes: Based on the high-dimensional continuous nature of the terrain parameters, a vector quantization variational autoencoder (VQ-VAE) is introduced for discretization representation to obtain discrete terrain parameter features. The discrete terrain parameter features are compressed, and a preset codebook maps the compressed features to the discrete token sequence.

[0008] Furthermore, the multimodal encoder adopts a branch-parallel structure, sets up dedicated encoding paths for the characteristics of different modal data, and extracts the core features of each modality; The MobileNet encoder extracts the spatial correlation of deformation between adjacent monitoring points in the InSAR deformation data, as well as the deformation evolution trend at different time points, and outputs a tensor containing spatiotemporal coupling features. The long short-term memory network encoder extracts the long-term dependencies of meteorological factors in the meteorological data and outputs the temporal feature vectors of the meteorological factors. The sparse Transformer encoder extracts the spatial correlation between different terrain factors in the terrain parameters and transforms them into a structured feature discrete token sequence.

[0009] Furthermore, the cross-modal fusion is achieved by establishing an association mapping between different modal features through a cross-attention mechanism combined with a dynamic query selection mechanism, thereby obtaining a high-level semantic representation that integrates multimodal spatiotemporal information and outputting the fused feature containing spatiotemporal coupling information.

[0010] Furthermore, the prediction decisions of the GRPO reinforcement learning include the following: An offline reinforcement learning framework is used to construct an experience replay pool based on the historical landslide records and generate pseudo samples to supplement sparse events. By setting up a group relative strategy optimization and a dynamic reward mechanism, the prediction strategy is continuously optimized to adapt to the nonlinear dynamic characteristics in complex geological environments and to obtain real-time geological environment status to dynamically adjust the landslide probability prediction strategy. Based on the fusion features, the landslide probability prediction strategy is dynamically adjusted according to the real-time geological environment, and a landslide probability prediction decision adapted to the dynamic geological environment is output.

[0011] Furthermore, the calculation expression for the landslide probability prediction decision is as follows: ; in, For landslide probability prediction decision, the value range is [0, 1]; σ is the weight parameter; DynamicAdjustment(s) = sigmoid(InSAR_Rate⋅Rainfall); σ is the sigmoid function; h is the fused feature, and W and b are the weights and biases of the linear transformation.

[0012] Furthermore, the calculation expression for the group relative strategy optimization is as follows: ; in, The relative policy probability of the group; η=0.7 is the weight parameter; G represents the relative advantage of a group; G represents a sample group with similar geological conditions; Q() is the Q-value function.

[0013] Furthermore, the dynamic reward mechanism is implemented through a reward function, which is a weighted sum of prediction accuracy, risk penalty, and exploration reward. The calculation expression of the reward function is as follows: ; in, =0.6, =0.3, =0.1 is the weight parameter; p is the landslide probability output when verifying the multimodal landslide spatiotemporal prediction model, and its value ranges from [0, 1]. κ=10, threshold is the region adaptive threshold; H is the entropy of the prediction result.

[0014] Furthermore, the prediction accuracy characterizes the degree to which the predicted probability closely approximates the actual landslide situation; the risk penalty addresses underreporting in high-risk areas by employing a regional adaptive threshold, where a penalty is imposed when the actual landslide risk exceeds the regional threshold while the predicted value is lower than the threshold; the exploration reward is the entropy H of the prediction result.

[0015] The beneficial effects of this invention are as follows: Based on the "any-to-any" cross-modal interaction capability of the AnyGPT framework, this invention integrates multi-source information such as InSAR deformation data, meteorological factors, terrain parameters, and historical landslide records into a unified high-level semantic representation, effectively breaking down information barriers between modalities and enabling the model to capture spatiotemporal coupling patterns that are difficult to identify using traditional methods. The introduction of VQ-VAE transforms high-dimensional continuous terrain parameters into discrete token sequences, preserving core terrain features while significantly reducing data dimensionality and computational complexity. The combination of GRPO reinforcement learning and the AnyGPT framework allows the multimodal landslide spatiotemporal prediction model to dynamically adjust its prediction strategy based on real-time changes in the geological environment, guided by "any-to-any" cross-modal features. By setting a reward function, the multimodal landslide spatiotemporal prediction model's ability to identify high-risk areas and its risk control level are improved, providing more sufficient response time for geological disaster early warning and demonstrating the practical value of the technical solution in meeting actual disaster prevention and mitigation needs. Attached Figure Description

[0016] Figure 1 This is a flowchart of the construction method of the multimodal landslide spatiotemporal prediction model provided by the present invention; Figure 2 This is a diagram showing the comparative experimental results provided by the present invention. Detailed Implementation

[0017] The technical solution of the present invention is further described below, but the scope of protection is not limited to what is described.

[0018] This invention provides a method for constructing a multimodal landslide spatiotemporal prediction model, such as... Figure 1 As shown, it includes: Step S100: Prepare a multimodal landslide dataset; the multimodal landslide dataset includes InSAR deformation data, meteorological data, topographic parameters, and historical landslide records; Step S200: Preprocess the multimodal landslide dataset to generate a multimodal landslide sample set; The preprocessing includes the following: For the InSAR deformation data, the slope-direction deformation rate sequence is obtained using SBAS-InSAR technology. Although the InSAR deformation data can reflect the surface deformation trend, the displacement along the slope direction is directly related to landslides. Therefore, the line of sight is first projected onto the deformation rate along the slope direction to obtain slope-direction deformation rate data that better reflects the physical mechanism of landslides, allowing it to directly participate in subsequent spatiotemporal evolution analysis.

[0019] For the meteorological data, daily time-series data are converted into time-series data at a specified scale, including weekly or monthly scales, using a sliding window aggregation method. The slippage in daily time-series data such as rainfall, temperature, and humidity is often the result of long-term accumulation of meteorological factors. To capture this long-term impact, a sliding window aggregation method is used to convert daily data into weekly or monthly time-series data, preserving the temporal variation patterns of meteorological factors while reducing data redundancy and facilitating alignment with the time dimensions of other modalities.

[0020] For terrain parameters, a discrete token sequence is obtained by introducing a vector quantization variational autoencoder (VQ-VAE). The terrain parameters include slope, vegetation cover (NDVI), VS30 equivalent shear wave velocity, etc., which have high-dimensional continuous characteristics. Directly inputting them into the model will increase the computational burden and make it difficult to form a unified interaction with other modal data.

[0021] The process of obtaining the discrete token sequence includes: Based on the high-dimensional continuous nature of the terrain parameters, a vector quantization variational autoencoder (VQ-VAE) is introduced for discretization representation to obtain discrete terrain parameter features. The discrete terrain parameter features are compressed, and a preset codebook maps the compressed features to the discrete token sequence.

[0022] Specifically, a Vector Quantization Variational Autoencoder (VQ-VAE) is introduced for discretization representation: Terrain parameters are compressed using a convolutional neural network, and then a pre-defined codebook (codebook size set to 512, trained using the minimum reconstruction error criterion) is used to map the compressed features into a discrete token sequence. This not only achieves dimensionality reduction of high-dimensional features but also transforms continuous terrain features into discrete symbols that can interact with other modalities (such as discretized time-series data), providing a unified symbolic basis for the "any-to-any" cross-modal association of the AnyGPT framework. To quantify the reconstruction error and codebook mapping loss during the terrain parameter discretization process, the following loss function is used to optimize the VQ-VAE model: (1) in, α = 0.25, β = 0.5, γ = 0.25 are the loss function; α = 0.25, β = 0.5, γ = 0.25 are the hyperparameters. It is a continuous feature of the encoder output. These are the quantified features. This is to stop gradient operations; codebook is the default codebook.

[0023] Historical landslide records are tagged, with tags including the landslide location from the previous year and the activity intensity information for the preceding period. This information guides the model in learning the spatiotemporal patterns of landslide occurrences and provides a reference for designing reward mechanisms for reinforcement learning.

[0024] Step S300: Using the AnyGPT framework as the base model, the base model includes a multimodal encoder, a multimodal language model, and a multimodal decoder. The multimodal encoder is used to map data from different modalities to a common feature space, and includes a 3D convolutional neural network encoder, a long short-term memory network encoder, and a Transformer encoder. The multimodal language model is used to achieve cross-modal fusion and output fused features containing spatiotemporal coupling information. The multimodal decoder is used to output the final landslide probability prediction result. The core of the AnyGPT framework is to map different modal data to a common feature space through a unified encoding mechanism, and then capture cross-modal correlations through an interaction mechanism to achieve deep collaboration of InSAR deformation, meteorological, and topographic data.

[0025] Step S400: Replace the 3D convolutional neural network encoder in the base model with a MobileNet encoder, replace the Transformer encoder with a sparse Transformer encoder, set the GRPO reinforcement learning prediction decision between the multimodal language model and the multimodal decoder, and construct an improved model, wherein the improved model is an AnyGPT-SLIDE network structure. The MobileNet encoder is an InSAR branch used to process slope direction deformation rate sequences; the Long Short-Term Memory Network encoder is a meteorological branch used to process meteorological time-series data; and the Sparse Transformer encoder is a terrain branch used to process discrete token sequences of terrain parameters.

[0026] The multimodal encoder adopts a branch-parallel structure, sets up dedicated encoding paths for the characteristics of different modal data, and extracts the core features of each modality, ensuring that the core features of each modality are fully extracted.

[0027] The MobileNet encoder extracts the spatial correlation of deformation between adjacent monitoring points in the InSAR deformation data, as well as the deformation evolution trend at different time points, and outputs a tensor containing spatiotemporal coupling features; thus fully characterizing the spatiotemporal dynamic pattern of surface deformation.

[0028] The Long Short-Term Memory (LSTM) network encoder extracts the long-term dependencies of meteorological factors in the meteorological data through a gating mechanism (input gate, forget gate, output gate) and outputs the temporal feature vector of the meteorological factors. For example, the cumulative effect of continuous rainfall on surface water content, or the indirect effect of temperature changes on the strength of soil and rock. The output temporal feature vector not only retains the time series characteristics of the meteorological data, but also maintains consistency with the time dimension of the InSAR branch.

[0029] The sparse Transformer encoder extracts the spatial correlations between different terrain factors in the terrain parameters through a self-attention mechanism, transforming them into a structured sequence of discrete feature tokens. Since the discrete token sequence of terrain features is pre-aligned with the feature dimensions of other modalities, compatibility with subsequent cross-modal fusion is ensured.

[0030] The cross-modal fusion is key to achieving "any-to-any" interaction. It is achieved by establishing a mapping between different modal features through a cross-attention mechanism combined with a dynamic query selection mechanism, resulting in a high-level semantic representation that integrates multimodal spatiotemporal information, and outputting the fused features containing spatiotemporal coupling information. Specifically: when real-time monitoring shows rainfall intensity exceeding a preset threshold (e.g., daily rainfall ≥ 50 mm), the meteorological branch feature is automatically set as the "query"; when the InSAR deformation rate exceeds a threshold (e.g., weekly average deformation ≥ 3 mm), the InSAR branch feature is used as the query; in other scenarios, the InSAR feature is used as the query by default. Features from non-query modalities are concatenated as "keys" and "values," and attention weights are calculated to automatically focus the model on other features closely related to the current core factor—for example, in rainfall-dominated scenarios, the correlation between rainfall and terrain slope is strengthened; in deformation-dominated scenarios, the correlation between deformation rate and vegetation cover is strengthened. This mechanism breaks down information barriers between modalities, enabling dynamic "any-to-any" associations between InSAR deformation, meteorological, and topographic data within a unified feature space. The final output is a high-level semantic representation that integrates multimodal spatiotemporal information, providing comprehensive feature support for subsequent landslide probability prediction. The cross-attention weights based on the dynamic query selection mechanism are calculated as follows: (2) Where λ=0.3 is the weight parameter; Dynamic_Query(Q) is the output of the dynamic query selection mechanism (when the rainfall intensity is ≥50mm, Dynamic_Query(Q)=LSTM_Output; when the InSAR deformation rate is ≥3mm, Dynamic_Query(Q)=3D_CNN_Output; otherwise, Dynamic_Query(Q)=0). Q represents the dimension of the key; Q is the query; K is the key; V is the value; and the superscript T is for transpose.

[0031] The predictive decisions made by the GRPO reinforcement learning include the following: An offline reinforcement learning framework is used to construct an experience replay pool based on the historical landslide records and generate pseudo samples to supplement sparse events. By setting up a group relative strategy optimization and a dynamic reward mechanism, the prediction strategy is continuously optimized to adapt to the nonlinear dynamic characteristics in complex geological environments and to obtain real-time geological environment status to dynamically adjust the landslide probability prediction strategy. Based on the fusion features, the landslide probability prediction strategy is dynamically adjusted according to the real-time geological environment, and a landslide probability prediction decision adapted to the dynamic geological environment is output. This landslide probability prediction decision is the final landslide prediction result.

[0032] Landslide prediction is essentially a multi-stage decision-making problem in a dynamic environment, requiring adjustments to prediction strategies based on real-time changes in the geological environment. To address this, based on the multimodal fusion of the AnyGPT framework, the GRPO (Group Relative Policy Optimization) reinforcement learning algorithm is introduced. An offline reinforcement learning framework is employed, and an experience replay pool is constructed based on historical landslide records (using data augmentation to generate pseudo-samples to supplement sparse events). A dynamic reward mechanism is used to continuously optimize the prediction strategy, enabling the multimodal landslide spatiotemporal prediction model to adapt to the nonlinear dynamic characteristics of complex geological environments.

[0033] The core of GRPO reinforcement learning is to view the landslide prediction process as a cyclical interaction of "state-action-reward": "State" refers to the multimodal fusion features output by the AnyGPT framework, containing multi-dimensional information such as InSAR deformation, meteorology, and topography, comprehensively reflecting the overall state of the current geological environment; "Action" refers to the landslide probability prediction result output by the model, i.e., the judgment of the likelihood of a landslide occurring in a certain area in the future. This is used to guide policy optimization by fusing the relative advantages of the fusion group and the Q-value function.

[0034] The calculation expression for the group relative strategy optimization is: (3) in, The relative policy probability of the group; η=0.7 is the weight parameter; G represents the relative advantage of the group; G is a sample group with similar geological conditions (the similarity between the terrain parameter vector and the meteorological feature vector is calculated using cosine similarity, and the similarity threshold is set to ≥0.8); Q() is the Q-value function.

[0035] The goal of policy optimization is to enable the model to gradually learn better prediction policies through interaction with the environment (i.e., continuous prediction and adjustment based on actual results). To achieve this goal, GRPO introduces a "group relative advantage" evaluation mechanism: samples with similar geological conditions within the same batch (using cosine similarity to calculate the similarity between terrain parameter vectors and meteorological feature vectors, with a similarity threshold of ≥0.8) are grouped together. The merit of a prediction action is evaluated by calculating its relative reward within the group (the difference between the reward and the average reward of the group). This mechanism avoids the randomness of single-sample rewards in traditional reinforcement learning, making policy updates more stable, and is particularly suitable for policy transfer in similar geological environments. The policy gradient update formula combining group relative advantage and the advantage function is as follows: (4) Where α=0.001 is the learning rate, δ=0.5 is the weight parameter, A(s,a) is the advantage function, and t represents the gradient; These are the current policy network parameters. These are the updated policy network parameters. for The gradient.

[0036] The dynamic reward mechanism is implemented through a reward function, the design of which directly affects the learning direction of the model. The reward function is a weighted sum of prediction accuracy, risk penalty, and exploration reward. The calculation expression for the reward function is as follows: (5) in, =0.6, =0.3, =0.1 is the weight parameter; , The value of is for landslide probability prediction decision, and its range is [0, 1]; p is the landslide probability output when verifying the multimodal landslide spatiotemporal prediction model, and its range is [0, 1]. κ=10, threshold is the region adaptive threshold; H is the entropy of the prediction result.

[0037] The prediction accuracy characterizes how close the predicted probability is to the actual landslide situation; the model is encouraged to make the most accurate predictions possible. The risk penalty addresses underreporting in high-risk areas by employing a regional adaptive threshold (dynamically adjusted based on historical landslide frequency and population density, with a threshold of 0.6 for highly vulnerable areas and 0.8 for low-vulnerable areas). When the actual landslide risk exceeds the regional threshold while the predicted value is lower than the threshold, a penalty is imposed, forcing the model to focus on high-risk areas and reducing the risk of disaster underreporting. The exploration reward is the entropy H of the prediction result, which encourages the model to explore more regions with high prediction uncertainty (i.e., low confidence regions), avoids ignoring potential risks due to over-reliance on existing experience, and improves the model's generalization ability.

[0038] Through GRPO reinforcement learning, the model can dynamically adjust its prediction strategy based on real-time geological conditions, guided by multimodal fusion features. When accelerated InSAR deformation in a region is detected, accompanied by continuous rainfall, the model will combine high-slope features in the terrain to increase the prediction weight of landslide risk in that region. Conversely, the strategy will be adjusted accordingly when environmental conditions change (e.g., rainfall stops or deformation slows down). This dynamic optimization mechanism allows the static multimodal features output by the AnyGPT framework to be transformed into predictive decisions adapted to dynamic environments, compensating for the shortcomings of traditional supervised learning in dynamic strategy adjustment.

[0039] The calculation expression for the landslide probability prediction decision is as follows: (6) in, For landslide probability prediction decision, the value range is [0, 1]; σ is the weight parameter; DynamicAdjustment(s) = sigmoid(InSAR_Rate⋅Rainfall); σ is the sigmoid function; h is the fused feature, and W and b are the weights and biases of the linear transformation.

[0040] Step S500: Classify the multimodal landslide sample set into a training set, a validation set, and a test set; train and validate the improved model to generate a multimodal landslide spatiotemporal prediction model based on AnyGPT-SLIDE. Tested at 1000km 2 At the regional scale, the inference time of the multimodal landslide spatiotemporal prediction model can be controlled within 5 seconds, meeting the requirements for real-time early warning. The formula for the lightweight feature extraction-based warning model is as follows: (7) Where κ=0.1 is the weight parameter, SparseAttention(x) is a sparse attention mechanism that only focuses on Top-K relevant terrain factors (K=16), Conv(x) is a standard convolution, and DepthwiseConv(x) is a depthwise separable convolution.

[0041] The multimodal landslide spatiotemporal prediction model based on AnyGPT-SLIDE provided by this invention not only leverages the advantages of the AnyGPT framework in unified multimodal representation and cross-modal correlation, but also achieves policy adaptation in dynamic environments through GRPO reinforcement learning. This enables the model to efficiently capture the complex spatiotemporal correlation between InSAR deformation and meteorological and topographical factors, and to adjust the prediction strategy according to real-time changes in the geological environment, ultimately achieving high-precision and highly robust landslide spatiotemporal prediction. It successfully applies the "any-to-any" concept and the AnyGPT framework to the field of landslide prediction, breaking down the modal barriers of multi-source heterogeneous data. At the same time, it improves the adaptability to dynamic environments through reinforcement learning, providing an efficient and reliable technical path for landslide risk prediction under complex geological conditions.

[0042] Specific examples are as follows: First, multi-source data is collected and preprocessed to lay the foundation for "any-to-any" cross-modal interaction in the AnyGPT framework. For InSAR deformation data, Sentinel-1A ascending and descending orbit images are used. After processing with SBAS-InSAR to obtain the line-of-sight deformation rate, it is projected onto the slope direction to form deformation characteristics that better reflect the actual landslide. Meteorological data, such as daily values ​​of rainfall, temperature, and humidity, are selected and aggregated into weekly or monthly scale data through a sliding window to capture the cumulative impact of long-term meteorological factors. Topographic parameters, such as slope, vegetation cover, and VS30 equivalent shear wave velocity, are transformed into discrete token sequences using VQ-VAE (codebook size 512, minimum reconstruction error training). High-dimensional features are first compressed using a convolutional neural network, and then mapped to discrete symbols using a pre-set codebook, so that continuous topographic features can form a unified interactive form with other modal data. At the same time, historical landslide records, including the landslide location of the previous year and the activity intensity of the previous period, are collected as a reference for model learning and used to build an offline reinforcement learning experience replay pool (generating pseudo samples through data augmentation). After preprocessing, multimodal data from different sources and in different formats are transformed into standardized features adapted to the AnyGPT framework, preparing for "any-to-any" cross-modal information fusion.

[0043] Secondly, a multimodal spatiotemporal fusion model under the AnyGPT framework is constructed to achieve "any-to-any" cross-modal correlation. The spatiotemporal encoder of the model adopts a branch parallel structure, and a dedicated encoding path is designed for the characteristics of different modal data: the InSAR branch processes the temporal deformation data through MobileNet (replacing 3D CNN), extracts coupled features containing time and spatial dimensions, and outputs feature tensors that can reflect the dynamic evolution of surface deformation; the meteorological branch uses a long short-term memory network to process the aggregated meteorological time-series data, captures the long-term dependence of factors such as rainfall and temperature through a gating mechanism, and outputs a temporal feature vector compatible with other modal dimensions; the terrain branch inputs the discrete token sequence generated by VQ-VAE into the sparse Transformer encoder (focusing only on Top-K correlation factors), and uses a self-attention mechanism to model the spatial correlation between terrain factors to form a structured terrain feature sequence. Subsequently, the cross-modal fusion module establishes associations between different modal features through a cross-attention mechanism combined with dynamic query selection: when rainfall intensity is ≥50mm, meteorological features are used as queries; when InSAR deformation rate is ≥3mm, InSAR features are used as queries; and in other scenarios, InSAR is used as the default query. Features from non-query modalities are concatenated as keys and values, enabling the model to automatically focus on meteorological and topographic factors closely related to the current core factors, such as the superimposed effects of heavy rainfall and high-slope terrain. This integrates multimodal data into a unified high-level semantic representation, fully demonstrating the "any-to-any" cross-modal interaction capability of the AnyGPT framework.

[0044] Subsequently, GRPO reinforcement learning was introduced for policy optimization, allowing the model to adapt to the dynamic geological environment under the guidance of the fused features output by the AnyGPT framework. An offline reinforcement learning framework was adopted, using multimodal fused features as state input and landslide probability prediction results as action output, with a policy network realizing the mapping from state to action. In policy optimization, samples with similar geological states within the same batch (cosine similarity of terrain and meteorological feature vectors ≥ 0.8) were grouped, and the relative advantage of a certain action within the group was calculated to avoid the randomness of single-sample rewards and make policy updates more stable. The reward function consists of three parts: a base reward based on prediction accuracy, encouraging the model to make accurate predictions; a risk penalty using a regional adaptive threshold (0.6 for high-vulnerability areas and 0.8 for low-vulnerability areas), penalizing missed predictions in high-risk areas to ensure the model focuses on high-risk areas; and an exploration reward of the entropy value of the prediction result, encouraging the model to explore low-confidence areas and improve generalization ability. Through the dynamic reward mechanism of GRPO, the model can adjust the prediction strategy according to the real-time changes in the geological environment. For example, when InSAR deformation acceleration is detected and accompanied by continuous rainfall, the model can automatically improve the landslide risk assessment of the area, thus transforming the static fusion features of the AnyGPT framework into a prediction strategy that adapts to the dynamic environment.

[0045] Finally, the model was trained and validated. The experiment selected the reservoir-dam group section of the upper Yellow River and a typical mountainous area in Yunnan as the study area. The dataset contained 57 and 123 labeled landslide events, respectively, and was divided into training, validation, and test sets in a 7:2:1 ratio. The Adam optimizer was used, with a learning rate of 1e-4, 50 training epochs, and a batch size of 32. Performance was evaluated by averaging the results of 10 independent experiments. Comparative experiments included: comparing with traditional models (LSTM, Transformer) to verify accuracy improvement; comparing with CLIP to verify the advantages of multimodal fusion; comparing with PPO to verify the policy optimization effect of GRPO; and ablation experiments to verify the necessity of VQ-VAE and the GRPO module. The results show that, as Figure 2 As shown, the model achieved an AUC value of 0.93 in landslide prediction in both regions, a significant improvement over traditional models, and also within a 1000km range. 2 The inference time for the region is ≤5 seconds, meeting practical engineering requirements. Through the above implementation process, the AnyGPT-SLIDE model, leveraging the "any-to-any" concept and the AnyGPT framework, achieves deep fusion and dynamic strategy optimization of multi-source heterogeneous data, providing a practical technical solution for landslide prediction under complex geological conditions. Figure 2 (a) is a bar chart comparing the performance of this model with traditional models such as CLIP, Transformer, and LSTM. Figure 2(b) is a line graph showing the relationship between the model and response time at different regional scales; Figure 2 (c) is a radar chart showing the advantages of this model in a multi-dimensional evaluation of the comprehensive capabilities of the traditional model.

[0046] The multimodal landslide spatiotemporal prediction model based on AnyGPT-SLIDE provided by this invention significantly improves prediction accuracy and reliability. Thanks to the "any-to-any" cross-modal interaction capability of the AnyGPT framework (combined with a dynamic query selection mechanism), multi-source information such as InSAR deformation data, meteorological factors, terrain parameters, and historical landslide records are integrated into a unified high-level semantic representation, effectively breaking down information barriers between modalities. This enables the model to capture spatiotemporal coupling patterns that are difficult to identify using traditional methods—for example, the synergistic effect of slope direction deformation rate and continuous rainfall, and the superimposed risks of high-slope terrain and insufficient vegetation cover. Experimental results show that in tests conducted in the reservoir-dam cluster of the upper Yellow River (57 landslide samples) and typical mountainous areas of Yunnan (123 landslide samples), the model's AUC value reached over 0.92, an improvement of 15%–22% compared to the traditional LSTM (AUC 0.78) and Transformer models (AUC 0.81), with the F1-Score increasing to 0.82, significantly outperforming the baseline model. Furthermore, compared to the general multimodal framework CLIP (AUC 0.85 after adaptation for landslide prediction tasks), this model demonstrates a more significant improvement in accuracy due to its customized design for geological data. These results prove that multimodal fusion achieved through the AnyGPT framework can provide more comprehensive feature support for landslide prediction, significantly reducing prediction bias caused by insufficient information from a single data source, especially excelling in predicting long-term evolution trends under complex geological conditions.

[0047] The processing efficiency and resource adaptability of multimodal data are significantly optimized. The AnyGPT framework's implementation of the "any-to-any" concept relies on efficient and unified representation of heterogeneous data, and the introduction of VQ-VAE provides crucial support for this: by transforming high-dimensional continuous terrain parameters into discrete token sequences (codebook size 512), not only are core terrain features preserved, but data dimensionality and computational complexity are also significantly reduced. Experimental data shows that removing the VQ-VAE module reduces model training speed by 34% and increases memory usage by 27%, while retaining the module reduces model parameters by 42% and improves inference speed by 65%. Combined with a lightweight module (MobileNet + Sparse Transformer), performance at 1000km... 2The inference time for the region is controlled within 5 seconds, meeting the requirements for real-time early warning. This means that in resource-constrained scenarios (such as real-time early warning on mobile devices or large-scale regional monitoring), the model can still operate efficiently while maintaining high prediction accuracy (AUC decreases by only 3.1%). This efficiency stems from the AnyGPT framework's lightweight processing capabilities for multimodal data, enabling it to adapt to both high-dimensional and complex geological data and meet the economical computational resource requirements of practical engineering.

[0048] The AnyGPT-SLIDE-based multimodal landslide spatiotemporal prediction model significantly enhances its adaptability and generalization ability to dynamic geological environments. The combination of GRPO reinforcement learning (offline framework + group relative advantage mechanism) and the AnyGPT framework enables the model to dynamically adjust its prediction strategy based on real-time changes in the geological environment, guided by "any-to-any" cross-modal features. For example, when InSAR data detects accelerated deformation (weekly average deformation ≥3mm) in a region, and meteorological data shows continuous heavy rainfall (daily rainfall ≥50mm), the model strengthens the correlation between these two types of features through a cross-attention mechanism and, guided by the GRPO reward mechanism, increases the weight of landslide risk assessment for that region. For new geological scenarios not found in historical data (such as combinations of special terrain and extreme weather), the model actively focuses on low-confidence areas through an exploration reward mechanism, avoiding the over-reliance on experience and the neglect of potential risks. Ablation experiments demonstrate that removing the GRPO module reduces the model's AUC by 8.2% and the F1-Score to 0.75. Compared to the PPO reinforcement learning algorithm, GRPO is superior in policy stability (reducing AUC fluctuation during training by 40%) and convergence speed (reducing the number of iterations to reach stable accuracy by 25%). This indicates that the group relative advantage evaluation mechanism effectively solves the problem of inefficient policy optimization caused by reward sparsity in traditional reinforcement learning, enabling the model to maintain stable prediction performance in dynamic environments, especially in generalization tasks across regions and geological conditions.

[0049] Furthermore, the model significantly improves its ability to identify high-risk areas and its risk control level. The design of the "risk penalty" (regional adaptive threshold) in the reward function, combined with the AnyGPT framework's sensitive capture of multimodal features, enables the model to accurately locate high-risk areas: when InSAR deformation data shows an abnormal displacement rate in a certain area and terrain parameters indicate a steep slope, the model prioritizes these high-risk points with "deformation-terrain" coupling. Even if meteorological data does not yet show extreme conditions, it will remain vigilant through reinforcement learning strategies to reduce the risk of missed reports. In actual testing, the missed report rate for high-risk areas decreased by approximately 30%, providing more sufficient response time for geological disaster early warning and demonstrating the practical value of the technical solution in meeting actual disaster prevention and mitigation needs.

[0050] The above-disclosed embodiments are merely specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for constructing a multimodal landslide spatiotemporal prediction model, characterized in that, include: Prepare a multimodal landslide dataset; the multimodal landslide dataset includes InSAR deformation data, meteorological data, topographic parameters, and historical landslide records; The multimodal landslide dataset is preprocessed to generate a multimodal landslide sample set; The AnyGPT framework is used as the base model, which includes a multimodal encoder, a multimodal language model, and a multimodal decoder. The multimodal encoder is used to map data from different modalities to a common feature space, and includes a 3D convolutional neural network encoder, a long short-term memory network encoder, and a Transformer encoder. The multimodal language model is used to achieve cross-modal fusion and output fused features containing spatiotemporal coupling information. The multimodal decoder is used to output the final landslide probability prediction result; The 3D convolutional neural network encoder in the base model is replaced with a MobileNet encoder, the Transformer encoder is replaced with a sparse Transformer encoder, and GRPO reinforcement learning prediction decision is set between the multimodal language model and the multimodal decoder to construct an improved model, which is an AnyGPT-SLIDE network structure. The multimodal landslide sample set is classified into a training set, a validation set, and a test set. The improved model is then trained and validated to generate a multimodal landslide spatiotemporal prediction model based on AnyGPT-SLIDE. The MobileNet encoder is an InSAR branch used to process slope direction deformation rate sequences; the Long Short-Term Memory Network encoder is a meteorological branch used to process meteorological time-series data; and the Sparse Transformer encoder is a terrain branch used to process discrete token sequences of terrain parameters.

2. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 1, characterized in that, The preprocessing includes the following: For the InSAR deformation data, the slope direction deformation rate sequence is obtained using SBAS-InSAR technology; For the meteorological data, the daily time-series data in the meteorological data is converted into time-series data at a specified scale by means of sliding window aggregation, and the specified scale includes weekly scale or monthly scale; For terrain parameters, a discrete token sequence is obtained by introducing a vector quantization variational autoencoder (VQ-VAE). Historical landslide records are tagged, with the tags including the landslide location from the previous year and the activity intensity information for the previous period.

3. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 2, characterized in that, The process of obtaining the discrete token sequence includes: Based on the high-dimensional continuous nature of the terrain parameters, a vector quantization variational autoencoder (VQ-VAE) is introduced for discretization representation to obtain discrete terrain parameter features. The discrete terrain parameter features are compressed, and a preset codebook maps the compressed features to the discrete token sequence.

4. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 1, characterized in that, The multimodal encoder adopts a branch-parallel structure, sets up dedicated encoding paths for the characteristics of different modal data, and extracts the core features of each modality; The MobileNet encoder extracts the spatial correlation of deformation between adjacent monitoring points in the InSAR deformation data, as well as the deformation evolution trend at different time points, and outputs a tensor containing spatiotemporal coupling features. The long short-term memory network encoder extracts the long-term dependencies of meteorological factors in the meteorological data and outputs the temporal feature vectors of the meteorological factors. The sparse Transformer encoder extracts the spatial correlation between different terrain factors in the terrain parameters and transforms them into a structured feature discrete token sequence.

5. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 1, characterized in that, The cross-modal fusion is achieved by establishing an association mapping between different modal features through a cross-attention mechanism combined with a dynamic query selection mechanism, thereby obtaining a high-level semantic representation that integrates multimodal spatiotemporal information and outputting the fused feature containing spatiotemporal coupling information.

6. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 1, characterized in that, The predictive decisions made by the GRPO reinforcement learning include the following: An offline reinforcement learning framework is used to construct an experience replay pool based on the historical landslide records and generate pseudo samples to supplement sparse events. By setting up a group relative strategy optimization and a dynamic reward mechanism, the prediction strategy is continuously optimized to adapt to the nonlinear dynamic characteristics in complex geological environments and to obtain real-time geological environment status to dynamically adjust the landslide probability prediction strategy. Based on the fusion features, the landslide probability prediction strategy is dynamically adjusted according to the real-time geological environment, and a landslide probability prediction decision adapted to the dynamic geological environment is output.

7. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 6, characterized in that, The calculation expression for the landslide probability prediction decision is as follows: ； in, For landslide probability prediction decision, the value range is [0, 1]; σ is the weight parameter; DynamicAdjustment(s) = sigmoid(InSAR_Rate⋅Rainfall); σ is the sigmoid function; h is the fused feature, and W and b are the weights and biases of the linear transformation.

8. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 6, characterized in that, The calculation expression for the group relative strategy optimization is: ； in, The relative policy probability of the group; η=0.7 is the weight parameter; G represents the relative advantage of a group; G represents a sample group with similar geological conditions; Q() is the Q-value function.

9. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 7, characterized in that, The dynamic reward mechanism is implemented through a reward function, which is a weighted sum of three parts: prediction accuracy, risk penalty, and exploration reward. The calculation expression of the reward function is as follows: ； in, =0.6, =0.3, =0.1 is the weight parameter; p is the landslide probability output when verifying the multimodal landslide spatiotemporal prediction model, and its value ranges from [0, 1]. κ=10, threshold is the region adaptive threshold; H is the entropy of the prediction result.

10. The method for constructing a multimodal landslide spatiotemporal prediction model as described in claim 9, characterized in that, The prediction accuracy characterizes how close the predicted probability is to the actual landslide situation; the risk penalty targets the underreporting of high-risk areas, and adopts a regional adaptive threshold. When the actual landslide risk exceeds the regional threshold while the predicted value is lower than the threshold, a penalty is imposed. The exploration reward is the entropy H of the prediction result.

Images