A forest fire spread and backtracking prediction method based on multi-modal deep learning

Abstract

The present application relates to a kind of forest fire spread and backtracking prediction method based on multi-modal deep learning, comprising the following steps: data collection and pretreatment;Using SwinTransformer image encoder module and the sequence encoder module of multi-head self-attention mechanism, respectively extract spatial features F spatial And time characteristics F temporal ;Utilize the environmental guidance area representation learning module consisting of cross attention mechanism and feedforward layer, learn the correlation between fire area and spatial environment data and time environment data, generate the representation of historical fire area in combination with spatial and time dimensions;Based on historical fire area representation A, learn the time dependence of target fire area, obtain the representation of target fire area;Using SwinTransformer image decoder module, the spread prediction or backtracking prediction of target fire area representation R is carried out, and target fire area T is output.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and computer vision, and relates to image prediction technology, in particular to a method for realizing forest fire spread and backtracking prediction through multi-modal deep learning. BACKGROUND

[0002] With the intensification of global climate change and the frequent occurrence of extreme weather events, the frequency and scale of forest fires have significantly increased, causing serious impacts on the ecological environment and human society, and fire extinguishing has become a difficult task. Accurate and real-time forest fire spread prediction is crucial for organizing evacuation and directing rescue operations. On the other hand, forest fire backtracking helps identify high-risk ignition areas, helping people prevent potential fires. Traditional forest fire prediction and backtracking methods mainly rely on physical model-based and experience rule-based methods, such as simulating the impact of wind speed, humidity, and temperature on fire spread. However, these methods often have insufficient accuracy and poor adaptability when dealing with complex environmental factors, making it difficult to accurately predict the spread path of the fire or trace the origin of the fire.

[0003] With the development of computer vision and deep learning technology, data-driven methods have gradually been applied to forest fire spread prediction and backtracking prediction. These models learn fire spread patterns from a large amount of data on environmental factors and forest fire areas over time. However, existing deep learning models heavily rely on large-scale labeled data during training, while real fire data is difficult to obtain, has limited coverage, and is unevenly distributed, making insufficient training data a major bottleneck. In addition, due to the complexity and uncertainty of environmental variables in real data, existing models often show significant gaps between simulated and real-world fire scenarios during training, resulting in poor performance of the model in actual application. SUMMARY

[0004] The present application provides a forest fire spread and backtracking prediction method based on multi-modal deep learning, which can effectively learn fire spread patterns by utilizing the spatio-temporal features of multi-modal data, and effectively improve the prediction accuracy of forest fire spread paths and the backtracking ability of fire origins by combining multi-modal environmental data and deep learning technology. This method uses virtual simulation fire data and a small amount of real fire data for training, which can reduce the performance gap between simulated data and real data, and significantly improve the performance of the model in actual application.

[0005] The present application is implemented by the following technical solutions:

[0006] A forest fire spread and backtracking prediction method based on multi-modal deep learning, comprising the following steps:

[0007] S1, data collection and preprocessing: collect multi-modal environmental information of the study area during the fire occurrence period, including terrain, vegetation coverage, fuel distribution map, weather information and remote sensing satellite image sequence, standardize and label the data, and obtain historical fire area image sequence I s , target fire area image sequence I t , remote sensing satellite image sequence I satellite , multi-modal time environment data X temporal , and multi-modal spatial environment data X spatial , X spatial ={X topography , X vegetation , X fuel}); wherein the multi-modal time environment data X temporal =X weather , X weather is the weather data of the simulated ignition point of the study area, the multi-modal spatial environment data X spatial ={X topography , X vegetation , X fuel}, X topography , X vegetation , X fuel are the terrain, vegetation coverage and fuel distribution data of the simulated ignition point of the study area; the historical fire area image sequence I s , the multi-modal time environment data X temporal , and the multi-modal spatial environment data X spatial are used as the input of the model, and the fire area sequence at the target time is predicted by the model, and the target fire area image sequence I t is used as supervision.

[0008] S2, a sequence encoder module composed of a Swin Transformer image encoder module and a multi-head self-attention mechanism is used to extract spatial features F spatial and temporal features F temporal from the input data, respectively.

[0009] S3, an environment-guided region representation learning module composed of a cross-attention mechanism and a feedforward layer is used to learn the correlation between the fire area and the spatial environment data and the time environment data, and to generate the representation A of the historical fire area in combination with the spatial and temporal dimensions.

[0010] S4, the time series fire area regression module includes a self-attention layer and a feedforward layer, which learns the time dependence of the target fire area based on the historical fire area representation A, and obtains the representation R of the target fire area to adapt to the time distribution interval of the target fire area.

[0011] S5, using the Swin Transformer image decoder module, performing spread prediction or backtracking prediction on the target fire area representation R, and outputting a target fire area T;

[0012] S6, training a deep neural network model on the fire scene data generated by the simulator and verifying the model performance on real fire scene data, and adjusting the network parameters to reduce the gap between the simulation data and the real data.

[0013] Further, the data collection and preprocessing of S1 includes the following steps:

[0014] S11, collecting multi-modal environmental information of the study area during the fire occurrence period, including terrain, vegetation cover, fuel distribution map, weather information and remote sensing satellite image sequence;

[0015] S12, generating a fire area sequence:

[0016] Creating a fire area sequence for training: using the Farsite model to simulate fires according to the environmental information of the study area, selecting a fire point for the study area, generating a simulated fire sequence for each fire point, and sampling a historical fire area image sequence and a target fire area image sequence for each simulated fire sequence, represented by I s and I t respectively;

[0017] Creating a fire area sequence for verification: sampling a historical fire area image sequence and a target fire area image sequence according to the remote sensing satellite image sequence of the study area;

[0018] S13, standardizing and labeling the data

[0019] Standardizing terrain, vegetation cover, fuel distribution and weather data to avoid bias caused by different data scales during model training;

[0020] Atmospheric correction, geometric correction and radiation correction are performed on the remote sensing satellite images to eliminate atmospheric and sensor effects; the corrected images are multispectral synthesized to derive RGB images;

[0021] Fire area labeling: according to the fire data simulated using the Farsite model, the fire occurrence area in the remote sensing satellite image sequence of the study area is labeled; the labeling result is represented in the form of binary fire mask, i.e. each pixel value is 1 indicating fire area and 0 indicating non-fire area;

[0022] S14, making training set and verification set

[0023] The training set comes from the fire area sequence for training after standardization and labeling;

[0024] The verification set comes from the standardized and annotated fire area sequence for verification;

[0025] Historical fire area image sequence I s , multi-modal time environment data X temporal (where X temporal = X weather ), X weather is the weather data of the simulated ignition point of the study area, multi-modal spatial environment data X spatial (where X spatial = {X topography , X vegetation , X ffuel}), X topography , X vegetation , X fuel are respectively the terrain, vegetation cover, and fuel distribution data of the simulated ignition point of the study area as input to the model, and the fire area sequence at the target time is predicted by the model, with the target fire area image sequence I t as supervision.

[0026] Further, S2 uses a sequence encoder module composed of a Swin Transformer image encoder module and a multi-head self-attention mechanism to extract spatial features F spatial and temporal features F temporal from the input data respectively. The specific method is as follows:

[0027] S21, using a Swin Transformer image encoder to extract features F s of the historical fire area image sequence I s ;

[0028] S22, using a Swin Transformer image encoder to extract features of the image format spatial environment data X spatial and the remote sensing satellite image sequence X satellite , and splicing them to obtain the spatial features F spatial ;

[0029] S23, using a sequence encoder module composed of a multi-head self-attention mechanism to extract features of the sequence format time environment data X temporal , and splicing it with the features of the remote sensing satellite image sequence X satellite to obtain the temporal features F temporal .

[0030] Further, the method of constructing the Swin Transformer image encoder module is as follows: the Swin Transformer encoder is used to extract multi-scale spatial features of input multi-modal data, including a plurality of stage modules StageModule, each of which extracts multi-scale features of an input image at different resolutions; the Swin Transformer image encoder module includes three main stages; the first stage receives an input image and converts the channel number thereof to a hidden dimension, while reducing the resolution of the image to one quarter of the original size through a downsampling operation; the input image is processed through a plurality of SwinBlock and convolution layers to extract preliminary spatial features; the second stage further processes the output of the first stage, the input channel number of this stage is increased, the hidden dimension is doubled, and the resolution of the image is halved again; through deeper SwinBlock and convolution layers, more rich mesoscale spatial features are extracted; the third stage is used to extract the highest level of features, in this stage, the input channel number and the hidden dimension are increased again, and the resolution of the image is further halved; the feature map of this stage contains the deepest level of information in the input image.

[0031] Further, step S3 uses an environment-guided region representation learning module composed of a cross-attention mechanism and a feedforward layer to learn the correlation between the fire region and the spatial and temporal environment data, and the specific method of generating the representation A of the historical fire region in combination with the spatial and temporal dimensions is as follows:

[0032] (1) Through the cross-attention mechanism, the feature F of the historical fire region image sequence I is taken as Q, and the spatial feature F of the multi-modal data is taken as K and V to obtain the representation A of the fire region and the spatial environment data. s s spatial spatial ;

[0033] (2) Through the cross-attention mechanism, the feature F of the historical fire region image sequence I is taken as Q, and the temporal feature F of the multi-modal data is taken as K and V to obtain the representation A of the fire region and the temporal environment data. s s temporal temporal ;

[0034] (3) The representation A of the spatial environment data and the representation A of the temporal environment data are fused to generate the representation A of the historical fire region. spatial temporal

[0035] ​​​​​​​​Further, in step S6, Binary Dice Loss and Binary Focal Loss are used as loss functions to optimize the neural network parameters θ by gradient descent method; AUPRC, IOU and F1 score are used as evaluation indexes to evaluate the prediction accuracy and robustness of the model.

[0036] The beneficial effects of the technical solutions provided by the present application are:

[0037] 1、The present application can consider the influence of various environmental factors on fire spread through multi-modal data fusion, and realize accurate prediction of forest fire spread path and accurate backtracking of origin.

[0038] 2、The present application adopts cross-attention mechanism and time series regression technology, and performs cross-domain feature extraction and fusion on multi-modal data through deep learning, which significantly improves the fire prediction and backtracking ability of the model in complex environment.

[0039] 3、The present application generates diversified training samples by combining different fire starting points, fuel types and simulated scene data of meteorological conditions in the model training process, which improves the adaptability of the model to complex fire scenes, and has strong practicality and popularization value. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a forest fire spread and backtracking prediction method based on multi-modal deep learning;

[0041] Figure 2 A precision comparison between the method of the present application and existing spatio-temporal data prediction methods; DETAILED DESCRIPTION

[0042] The technical solutions in the present application will be described clearly and completely below in combination with the drawings and embodiments. Based on the technical solutions in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] (I) Data collection and preprocessing, preparing multi-modal fire spread data set for subsequent forest fire spread and backtracking prediction model training and verification, the specific steps are as follows:

[0044] Select the research area, rely on Google Earth Engine platform to screen large fires with fire area greater than 25000 hectares in GlobFire data set as research area.

[0045] Collect environmental data of the research area during the fire occurrence period, including terrain data, vegetation coverage data, fuel distribution data, weather information, remote sensing satellite image sequence

[0046] Terrain data, vegetation cover data, and fuel distribution data are collected from the LANDFIRE website: Terrain data refers to the Digital Elevation Model (DEM) of the study area, which reflects the relief changes of the terrain. Terrain data is stored in raster format with a resolution of 30 meters; Vegetation cover data refers to the vegetation cover information of the study area. Vegetation cover data is expressed in percentage form, ranging from 0% to 100%; Fuel distribution data refers to the fuel distribution map within the study area. The fuel distribution map usually includes different types of fuel, such as grass, shrubs, trees, etc., and their corresponding fuel load values. Fuel distribution data is stored in raster format, with each pixel value representing the fuel type and its density in that area.

[0047] Weather information is collected from the Remote Automatic Weather Station website, including key meteorological variables such as wind speed, wind direction, temperature, humidity, etc., with a time resolution of hourly level.

[0048] Remote sensing satellite image sequence: Obtain remote sensing satellite image data from Sentinel-2 or Landsat series satellites covering the study area. These images have multiple spectral bands for monitoring changes in the fire area.

[0049] Fire area sequence generation.

[0050] Create fire area sequences for training. According to the environmental information of the study area, use the Farsite model to simulate fires, randomly select 100 ignition points for each study area, corresponding to generate 100 sequences of simulated fires, for each sequence, sample the historical fire area image sequence and the target fire area image sequence, respectively represented by I s and I t . The method is: for a sequence, sample some fire area images I s with early arrangement order to form the historical fire area image sequence; sample some fire area images with late arrangement order to form the target fire area image sequence I t .

[0051] Create fire area sequences for verification. According to the remote sensing satellite image sequence of the study area, use the same method as above to sample the historical fire area image sequence and the target fire area image sequence, respectively represented by I s ′ and I t ′.

[0052] Data standardization and data annotation.

[0053] Terrain, vegetation cover, fuel distribution, and weather data standardization: Standardize terrain, vegetation cover, fuel distribution, and weather data. Standardization is done using the following formula:

[0054]

[0055] where X is the original data value, μ is the mean of the data, σ is the standard deviation, and X' is the standardized data. This step ensures that all input data has the same scale and distribution before model processing, avoiding bias caused by different data scales during model training.

[0056] Remote sensing satellite image preprocessing: Perform atmospheric correction, geometric correction, and radiometric correction on remote sensing satellite images to eliminate atmospheric and sensor effects. The corrected images are then multispectral synthesized to derive RGB images.

[0057] Fire area annotation: Annotate the fire occurrence areas in the remote sensing satellite image sequence of the study area based on the fire data simulated using the Farsite model. The annotation result is represented in the form of a binary fire mask, where each pixel value of 1 represents a fire area and a value of 0 represents a non-fire area.

[0058] Create training and validation sets.

[0059] Create training and validation sets based on simulated fire data and real fire data, respectively, for model training and validation. The organization of the training and validation sets is consistent, with the only difference being the source of the historical fire area image sequence I s and the target fire area image sequence I t . The historical fire area image sequence I s , the multi-modal temporal environmental data X temporal (where X temporal = X weather ), X weather is the weather data of the simulated ignition point in the study area, and the multi-modal spatial environmental data X spatial (where X spatial = {X topography , X vegetation , X fuel}, X topography , X vegetation , X fuel are the terrain, vegetation cover, and fuel distribution data of the simulated ignition point in the study area) are used as inputs to the model to predict the fire area sequence at the target time, so the target fire area image sequence I t is used as supervision.

[0060] (ii) using a sequence encoder module composed of a Swin Transformer image encoder module and a multi-head self-attention mechanism, to extract spatial features F spatial and temporal features F temporal from the input data, respectively. The specific steps are as follows:

[0061] (1) First, a Swin Transformer image encoder module (SwinTransformerEncoder) is built. This encoder is used to extract multi-scale spatial features of the input multi-modal data, and is composed of multiple stage modules (StageModule). Each stage module extracts multi-scale features of the input image at different resolutions. The design purpose of the Swin Transformer encoder is to extract meaningful features from the input multi-modal data, and to provide high-quality feature representations for subsequent prediction tasks.

[0062] The entire encoder contains three main stages. The first stage (Stage 1) receives the input image and converts its channel number to the hidden dimension, while reducing the resolution of the image to one quarter of the original size through downsampling operations. In this stage, the input image is processed through multiple SwinBlocks and convolutional layers to extract preliminary spatial features, and the output is a feature map with a size of 64x64 pixels. The second stage (Stage 2) further processes the output of Stage 1. The input channel number of this stage is increased, the hidden dimension is doubled, and the resolution of the image is halved again to generate a feature map with a size of 32x32 pixels. Through deeper SwinBlocks and convolutional layers, this stage extracts more rich mesoscale spatial features. The third stage (Stage 3) is the last stage of the encoder, responsible for extracting the highest level of features. In this stage, the input channel number and hidden dimension are increased again, and the resolution of the image is further halved to output a feature map with a size of 16x16 pixels. The feature map of this stage contains the deepest information in the input image, which is used for the final prediction of the model.

[0063] In each StageModule, the input data first passes through a Patch Merging layer, which is responsible for downsampling and increasing the number of channels. Subsequently, the data enters multiple SwinBlocks for processing. Each SwinBlock is composed of a multi-head self-attention mechanism and a feedforward network. The attention mechanism is used to capture different features in space, and the feedforward network further processes these features. The SwinBlocks in each StageModule work alternately in standard and displacement modes to capture more rich feature information.

[0064] Through multi-stage feature extraction and down-sampling operations, the Swin Transformer encoder can effectively extract multi-scale spatial features from the input data. These features will be used in subsequent complex tasks such as fire spread path prediction, providing strong feature representation capabilities.

[0065] Features F of the historical fire region image sequence I s are extracted using the Swin Transformer encoder. s The formula is as follows:

[0066] F s = Enc Swin (I s )

[0067] Where Enc Swin represents the Swin Transformer encoder.

[0068] Features of the image format spatial environment data X spatial and the remote sensing satellite image sequence X satellite are extracted using the Swin Transformer encoder, and are spliced to obtain spatial features F spatial The formula is as follows:

[0069] F spatial = [Enc Swin (X spatial ), Enc Swin (X satellite )]

[0070] Where [·] represents the splicing process.

[0071] Next, a sequence encoder module composed of multi-head self-attention mechanism (WxEncoder) is built to encode weather data. First, the input weather data is processed by the node feature extraction layer. This layer performs preliminary feature extraction on the data through a 1D convolutional layer, with the convolution kernel size set to 1 to ensure that only the channel dimension is transformed without affecting the time dimension. In this way, the input weather data is mapped to the hidden dimension (default is 128), and then the output of the convolutional layer is nonlinearly transformed by the ELU activation function to generate more rich node features.

[0072] Subsequently, the node features are input into the multi-head self-attention mechanism. This mechanism is composed of multiple attention heads and can calculate dependencies between different time steps of the time series. In the last step, the output of the attention layer is processed by a linear layer to further map the features to the specified output dimension, and again nonlinearly processed by the ELU activation function.

[0073] The sequence encoder module composed of multi-head self-attention mechanism is used to extract the time environment data X in sequence format temporal The features of the remote sensing satellite image sequence X satellite are spliced with the features of the remote sensing satellite image sequence X temporal ,

[0074] F temporal = [Enc Wea (X temporal ), Enc swin (X satellite )]

[0075] Wherein, Enc Wea represents a sequence encoder composed of multi-head self-attention mechanism.

[0076] (Three) Use the environment guide region representation learning module composed of cross attention mechanism and feedforward layer to learn the correlation between fire region and spatial environment data and time environment data, and generate the representation A of historical fire region combining spatial and time dimensions. The specific steps are as follows:

[0077] (1) Through the cross attention mechanism, the feature F s of the historical fire region image sequence I s is taken as Q, the spatial feature F spatial of the multi-modal data is taken as K and V, and the representation A spatial learning the correlation between fire region and spatial environment data is obtained.

[0078] (2) Through the cross attention mechanism, the feature F s of the historical fire region image sequence I s is taken as Q, the time feature F temporal of the multi-modal data is taken as K and V, and the representation A temporal learning the correlation between fire region and time environment data is obtained.

[0079] (3) The representation A spatial of the spatial environment data and the representation A temporal of the time environment data are fused to generate the representation A of the historical fire region.

[0080] (Four) The time series fire region regression module contains self-attention layer and feedforward layer, which learns the time dependence of the target fire region based on the historical fire region representation A, and obtains the representation R of the target fire region to adapt to the time distribution interval of the target fire region. The specific steps are as follows:

[0081] Time series modeling, the target fire region image sequence I ttimestamp encoding of the target fire region image sequence I

[0082]

[0083] wherein,

[0084] Q = W q pos([p(1),..., p(T)]), K = W k A, V = W v A

[0085] wherein, pos([p(1),..., p(T)] represents the timestamp encoding of the target fire region image sequence I t , which embeds the position of the image sequence into the query vector Q through the timestamp, and then associates it with the representation A of the historical fire region, capturing the dependency relationship in the time series.

[0086] In the implementation process, first, the input image features are mapped to the query space through the convolutional layer. Then, the query, key and value vectors are respectively passed to the multi-head attention mechanism for interactive calculation. The attention mechanism calculates the correlation weight between the query and the key, and then applies these weights to the value vector to generate the final attention output. The input image features are added element by element, and then normalized by layer normalization to ensure the stability of the model. Then, the features are further processed by the feedforward network, and normalized again to obtain the representation R of the target fire region.

[0087] (Five) Use the Swin Transformer image decoder module to perform spread prediction or backtracking prediction on the target fire region representation R, and output the target fire region T. The specific steps are as follows:

[0088] (1) Build a Swin Transformer image decoder module (SwinTransformerDecoder) to gradually restore the multi-scale features extracted by the encoder to a higher resolution, and finally output the prediction result of the target fire area. The Swin Transformer decoder module is composed of three upsampling stages, each of which gradually improves the resolution of the feature map and finally outputs the prediction result of the target fire area. In the first stage (Stage 4) of the decoder, the input is the feature map generated by the last stage of the encoder and the residual features of the intermediate layers of the encoder. This stage first increases the resolution of the feature map from 16x16 pixels to 32x32 pixels through upsampling operation, while reducing the number of channels to half of the input. This process aims to gradually restore image details. The second stage (Stage 5) further processes the output feature map from Stage 4 and combines the residual features generated by the first stage of the encoder. In this stage, the resolution of the feature map continues to increase to 64x64 pixels, while the number of channels is halved again. Through the processing of this stage, the decoder can gradually restore more rich image details. The last stage (Stage 6) is the final upsampling stage of the decoder, which is responsible for restoring the output feature map of Stage 5 to the original high resolution (256x256) and reducing the number of channels to 1 to output the final fire area prediction result. This stage uses fine upsampling operation to ensure spatial consistency and accuracy of the image at high resolution.

[0089] Using the Swin Transformer decoder to predict or backtrack the target fire area representation R, the formula for outputting the target fire area T is as follows:

[0090] T=Dec Swin (R)

[0091] where Dec Swin represents the Swin Transformer decoder.

[0092] (6) Train the deep neural network model on the virtual fire scene data generated by the simulator, and verify the model performance on real fire scene data. The specific steps are as follows:

[0093] Use Binary Dice Loss and Binary Focal Loss as loss functions to optimize the neural network parameters θ by gradient descent method.

[0094] Binary Dice Loss calculation formula:

[0095]

[0096] where p i represents the predicted value, g i represents the true value, and ∈ is a small constant to prevent division by zero.

[0097] Binary Focal Loss calculation formula:

[0098] FocalLoss = -α(1-p t ) γ log(p t )

[0099] where p t is the probability predicted by the model, and α and γ are adjustment terms used to control the weights of easy and difficult samples.

[0100] AUPRC (Area Under Precision-Recall Curve), IOU (Intersection over Union), and F1 score are used as evaluation indicators to evaluate the prediction accuracy and robustness of the model.

[0101] The feasibility of the method of the present application is verified by specific examples, as described below:

[0102] To verify the feasibility of the present application, we conducted comparative experiments on the Sim2Real-Fire fire prediction dataset, which consists of a large number of virtual simulation fire scenes and a small number of real fire scenes, covering various terrains, vegetation, fuel types, and weather conditions.

[0103] In the experiment, we compared the method of the present application with existing general spatio-temporal data prediction methods, and the evaluation indicators included AUPRC, IOU, and F1 score. The experimental results showed that the present application performed significantly better than existing methods in fire prediction and backtracking tasks in various complex environments. The specific experimental data and results are shown in the attached Figure 2 .

[0104] Through these experiments, the effectiveness and superiority of the method of the present application can be verified. The present application can effectively reduce the significant gap between simulated and real-world fire scenes, improve the prediction accuracy of fire spread paths and the backtracking ability of fire origins, and provide important technical support for forest fire warning and prevention.

Claims

1. A method for predicting the spread and retrospection of forest fires based on multimodal deep learning, comprising the following steps: S1, Data Collection and Preprocessing: Collect multimodal environmental information on the study area during the fire occurrence period, including topography, vegetation cover, fuel distribution maps, weather information, and remote sensing satellite image sequences. Standardize and label the data to obtain historical fire area image sequence I. s Image sequence I of the target fire area t Remote sensing satellite image sequence I satellite Multimodal temporal environmental data X temporal and multimodal spatial environment data X spatial X spatial ={X topography X vegetation X fuel }); where multimodal temporal environmental data X temporal =X weather X weather For the simulated ignition points in the study area, multimodal spatial environmental data X spatial ={X topography X vegetation X fuel }, X topography X vegetation X fuel The data includes topography, vegetation cover, and fuel distribution data of the simulated ignition point in the study area; and historical fire area image sequence I. s Multimodal temporal environmental data X temporal and multimodal spatial environment data X spatial The model takes the target fire area sequence as input and predicts the target time using the target fire area image sequence I. t As a form of oversight; S2 uses a sequence encoder module composed of a Swing Transformer image encoder module and a multi-head self-attention mechanism to extract spatial features F from the input data. spatial and time features F temporal ; S3 utilizes an environment-guided region representation learning module composed of a cross-attention mechanism and a feedforward layer to learn the correlation between fire areas and spatial and temporal environmental data. It then combines spatial and temporal dimensions to generate a representation A of historical fire areas. The method is as follows: (1) Through the cross-attention mechanism, historical fire area image sequence I s Feature F s As Q, the spatial feature F of multimodal data spatial As K and V, we obtain the representation A of the learned fire zone and spatial environment data. spatial ; (2) Through the cross-attention mechanism, historical fire area image sequence I s Feature F s As Q, the temporal feature F of multimodal data temporal As K and V, we obtain the representation A of the fire zone and temporal environmental data. temporal ; (3) Representation of integrated spatial environment data A spatial Representation of time and environmental data A temporal Generate a representation A of the historical fire area; S4, the time series fire area regression module includes a self-attention layer and a feedforward layer. Based on the historical fire area representation A, it learns the time dependence of the target fire area and obtains the target fire area representation R to adapt to the time distribution interval of the target fire area. The target fire zone is represented by R as follows: ； in, Q=W q ·pos([p(1),...,p(T)]),K=W k A,V=W v A Where pos([p(1), ..., p(T)] represents the target fire area image sequence I t The timestamp encoding embeds the location of the image sequence into the query vector Q through timestamps, and then associates it with the representation A of historical fire areas to capture the dependencies in the time series; S5 uses the Swin Transformer image decoder module to perform spread prediction or backtracking prediction on the target fire area representation R, and outputs the target fire area T. S6 trains a deep neural network model on fire scene data generated by the simulator, verifies the model performance on real fire scene data, and adjusts network parameters to reduce the gap between simulated and real data.

2. The forest fire spread and retrospective prediction method based on multimodal deep learning according to claim 1, characterized in that, S1 data collection and preprocessing includes the following steps: S11 collects multimodal environmental information on the time period of fire occurrence in the study area, including topography, vegetation cover, fuel distribution map, weather information and remote sensing satellite image sequence; S12, Generate the fire zone sequence: Creating fire zone sequences for training: Fires are simulated using the Farsite model based on environmental information of the study area. For each study area, ignition points are selected, and a simulated fire sequence is generated for each ignition point. For each simulated fire sequence, historical fire zone image sequences and target fire zone image sequences are sampled, and then processed using I... s and I t express; Create a sequence of fire areas for verification; sample historical fire area image sequences and target fire area image sequences based on remote sensing satellite image sequences of the study area; S13, Standardize and label the data. Standardize the terrain, vegetation cover, fuel distribution and weather data to avoid bias caused by different data scales during model training; Atmospheric, geometric, and radiometric corrections are performed on remote sensing satellite images to eliminate atmospheric and sensor effects; the corrected images are then multispectral synthesized to export RGB images. Fire area labeling: Based on fire data simulated using the Farsite model, fire-occurring areas in the remote sensing satellite image sequence of the study area are labeled; the labeling results are represented in the form of a binary fire mask, that is, each pixel value of 1 represents a fire area, and a value of 0 represents a non-fire area; S14, Creating the training and validation sets. The training set consists of standardized and labeled fire zone sequences used for training. The validation set consists of a standardized and labeled sequence of fire zones used for validation. Historical fire area image sequence I s Multimodal temporal environmental data X temporal (where X) temporal =X weather ), X weather For the simulated ignition points in the study area, multimodal spatial environmental data X spatial (where X) spatial ={X topography X vegetation X fuel }, X topography X vegetation X fuel The model takes as input the terrain, vegetation cover, and fuel distribution data of the simulated ignition point in the study area, and predicts the fire area sequence at the target time using the target fire area image sequence I. t As a supervisor.

3. The forest fire spread and retrospective prediction method based on multimodal deep learning according to claim 1, characterized in that, The method for S2 is as follows: S21, Use the Swin Transformer image encoder to extract historical fire area image sequence I s Feature F s ; S22, Use the Swing Transformer image encoder to extract spatial environment data X in image format. spatial and remote sensing satellite image sequence X satellite The features are then spliced ​​together to obtain the spatial feature F. spatial ; S23, extract the temporal context data X in sequence format using a sequence encoder module composed of a multi-head self-attention mechanism. temporal Features, and compared with remote sensing satellite image sequence X satellite The features are concatenated to obtain the time feature F. temporal .

4. The forest fire spread and retrospective prediction method based on multimodal deep learning according to claim 3, characterized in that, The method for constructing the Swin Transformer image encoder module is as follows: The Swin Transformer encoder is used to extract multi-scale spatial features of the input multimodal data, including multiple StageModules. Each StageModule extracts multi-scale features of the input image at different resolutions. The Swin Transformer image encoder module contains three main stages. The first stage receives the input image and converts its channel number into hidden dimensions. At the same time, it reduces the resolution of the image to one-quarter of the original size through downsampling operations. The input image is processed by multiple SwinBlocks and convolutional layers to extract preliminary spatial features. The second stage further processes the output of the first stage. In this stage, the number of input channels increases, the hidden dimension doubles, and the image resolution is halved again. Through deeper SwinBlock and convolutional layers, richer mesoscale spatial features are extracted. The third stage is used to extract the highest level features. In this stage, the number of input channels and hidden dimension increases again, and the image resolution is halved again. The feature map in this stage contains the deepest information in the input image.

5. The forest fire spread and retrospective prediction method based on multimodal deep learning according to claim 1, characterized in that, In step S6, Binary Dice Loss and Binary Focal Loss are used as loss functions, and the neural network parameters θ are optimized by gradient descent. AUPRC, IOU, and F1 score are used as evaluation metrics to assess the model's prediction accuracy and robustness.

Images