Forest fire spreading and backtracking prediction method based on multi-modal deep learning

Abstract

The invention relates to a forest fire spreading and backtracking prediction method based on multi-modal deep learning. The method comprises the following steps: data collection and preprocessing; a sequence encoder module composed of a SwinTransform image encoder module and a multi-head self-attention mechanism is used, and a spatial feature Fspatial and a time feature Ftemporal are respectively extracted from the input data; an environment guiding area representation learning module composed of a cross attention mechanism and a feed-forward layer is used for learning the correlation between the fire disaster area and the space environment data and the correlation between the fire disaster area and the time environment data, and representation of a historical fire disaster area is generated in combination with the space dimension and the time dimension; based on the historical fire area representation A, learning the time dependence of the target fire area to obtain the representation of the target fire area; and performing spreading prediction or backtracking prediction on the target fire area representation R by using a SwinTransform image decoder module, and outputting a target fire area T.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision, and relates to image prediction technology, specifically a method for realizing forest fire spread and backtracking prediction through multimodal deep learning. Background Art

[0002] With the intensification of global climate change and the frequent occurrence of extreme weather events, the frequency and scale of forest fires have increased significantly, bringing serious impacts on the ecological environment and human society, and fire fighting has become a difficult task. Accurate and real-time prediction of the spread of forest fires is crucial for organizing evacuations and directing rescue operations. On the other hand, forest fire backtracking helps identify high-risk ignition areas and helps people prevent potential fires. Traditional forest fire prediction and backtracking methods mainly rely on methods based on physical models and empirical rules, such as simulating the effects of factors such as wind speed, humidity and temperature on the spread of fire. However, these methods often have problems such as insufficient accuracy and poor adaptability when dealing with complex environmental factors, making it difficult to accurately predict the spread path of the fire or trace back the origin of the fire.

[0003] With the development of computer vision and deep learning technologies, data-driven methods have been gradually applied to the spread prediction and retrospective prediction of forest fires. These models learn fire spread patterns from a large amount of data on environmental factors and forest fire areas changing over time. However, existing deep learning models rely heavily on large-scale labeled data during training, while real fire data is difficult to obtain, has limited coverage, and is unevenly distributed, making insufficient training data a major bottleneck. In addition, due to the complexity and uncertainty of environmental variables in real data, existing models often show a significant gap between simulated and real-world fire scenarios during training, resulting in poor performance of the models in practical applications. Summary of the invention

[0004] The present invention provides a forest fire spread and backtracking prediction method based on multimodal deep learning. By utilizing the spatiotemporal characteristics of multimodal data, the present invention can effectively learn the fire spread pattern, and combine multimodal environmental data with deep learning technology to effectively improve the prediction accuracy of forest fire spread paths and the ability to backtrack the origin of fires. The method uses virtual simulated fire data and a small amount of real fire data for training, which can reduce the performance gap between simulated data and real data, and significantly improve the performance of the model in practical applications.

[0005] The present invention is achieved through the following technical solutions:

[0006] A forest fire spread and backtracking prediction method based on multimodal deep learning includes the following steps:

[0007] S1, data collection and preprocessing: Collect multimodal environmental information of the fire period in the study area, including terrain, vegetation coverage, fuel distribution map, weather information and remote sensing satellite image sequences, standardize and annotate the data, and obtain the historical fire area image sequence I s , target fire area image sequence I t , Remote Sensing Satellite Image Sequence I satellite , multimodal temporal environment data X temporal , and multimodal spatial environment data X spatial , X spatial ={X topography , X vegetation , X fuel}); where the multimodal temporal environment data X temporal =X weather , X weather The weather data of the simulated fire point in the study area and the multimodal spatial environmental data X spatial ={X topography , X vegetation , X fuel}, X topography , X vegetation , X fuel The terrain, vegetation coverage, and fuel distribution data of the simulated fire point in the study area are used as the image sequence of the historical fire area. s , multimodal temporal environment data X temporal , and multimodal spatial environment data X spatial As the input of the model, the fire area sequence at the target time is predicted by the model, and the target fire area image sequence I t as supervision;

[0008] S2, using the Swin Transformer image encoder module and the sequence encoder module composed of the multi-head self-attention mechanism, extracts spatial features F from the input data spatial and time feature F temporal ;

[0009] S3, using the environment-guided region representation learning module composed of a cross-attention mechanism and a feed-forward layer, learns the correlation between the fire area and the spatial and temporal environment data, and generates a representation A of the historical fire area by combining the spatial and temporal dimensions;

[0010] S4, the time series fire area regression module includes a self-attention layer and a feed-forward layer. Based on the historical fire area representation A, it learns the time dependency of the target fire area and obtains the representation R of the target fire area to adapt to the time distribution interval of the target fire area;

[0011] S5, using the Swin Transformer image decoder module, performs spread prediction or backtracking prediction on the target fire area representation R, and outputs the target fire area T;

[0012] S6, trains the deep neural network model on the fire scene data generated by the simulator, verifies the model performance on the real fire scene data, and adjusts the network parameters to reduce the gap between the simulated data and the real data.

[0013] Furthermore, the data collection and preprocessing of S1 includes the following steps:

[0014] S11, collect multimodal environmental information of the fire occurrence period in the study area, including topography, vegetation cover, fuel distribution map, weather information and remote sensing satellite image sequences;

[0015] S12, generate fire area sequence:

[0016] Create fire area sequences for training: Use the Farsite model to simulate fire according to the environmental information of the study area. For the study area, select the fire point, generate a simulated fire sequence for each fire point, and sample the historical fire area image sequence and the target fire area image sequence for each simulated fire sequence. s and I t express;

[0017] Create a fire area sequence for verification; sample historical fire area image sequences and target fire area image sequences based on the remote sensing satellite image sequence of the study area;

[0018] S 1 3, standardize and label the data

[0019] Standardize terrain, vegetation cover, fuel distribution, and weather data to avoid bias caused by different data scales during model training;

[0020] Perform atmospheric correction, geometric correction and radiation correction on remote sensing satellite images to eliminate atmospheric and sensor effects; perform multi-spectral synthesis on the corrected images and export RGB images;

[0021] Fire area labeling: Based on the fire data simulated by the Farsite model, the fire occurrence areas in the remote sensing satellite image sequence of the study area are labeled; the labeling results are expressed in the form of a binary fire mask, that is, each pixel value of 1 represents a fire area, and a value of 0 represents a non-fire area;

[0022] S14, create training set and validation set

[0023] The training set comes from the fire area sequences used for training that have been standardized and labeled;

[0024] The validation set comes from the fire area sequences that have been standardized and annotated for validation;

[0025] Historical fire area image sequence I s , multimodal temporal environment data X temporal (Where X temporal =X weather ), X weather The weather data of the simulated fire point in the study area and the multimodal spatial environmental data X spatial (Where X spatial ={X topography , X vegetation , X ffuel}, X topography , X vegetation , X fuel The terrain of the simulated fire point in the study area, vegetation coverage, and fuel distribution data are used as the input of the model. The model predicts the fire area sequence at the target time. The target fire area image sequence I t As supervision.

[0026] Furthermore, S2 uses a sequence encoder module composed of a Swin Transformer image encoder module and a multi-head self-attention mechanism to extract spatial features F spatial and time feature F temporal The specific method is:

[0027] S21, using Swin Transformer image encoder to extract historical fire area image sequence I s Features F s ;

[0028] S22, use Swin Transformer image encoder to extract spatial environment data in image format X spatial and remote sensing satellite image sequence X satellite The features of the spatial feature F are concatenated spatial ;

[0029] S23, using the sequence encoder module composed of a multi-head self-attention mechanism to extract the temporal environment data X in sequence format temporal and compare it with the remote sensing satellite image sequence X satellite The feature concatenation of temporal .

[0030] Furthermore, the method for constructing the Swin Transformer image encoder module is as follows: the Swin Transformer encoder is used to extract multi-scale spatial features of input multimodal data, including multiple stage modules StageModule, each stage module extracts multi-scale features of the input image at different resolutions; the Swin Transformer image encoder module includes three main stages; the first stage receives the input image and converts its number of channels into hidden dimensions, and at the same time reduces the resolution of the image to one-fourth of the original size through a downsampling operation. The input image is processed by multiple SwinBlocks and convolutional layers to extract preliminary spatial features; the second stage further processes the output of the first stage, the number of input channels in this stage increases, the hidden dimension doubles, and the resolution of the image is halved again; through deeper SwinBlocks and convolutional layers, richer mid-scale spatial features are extracted; the third stage is used to extract the highest level of features, in this stage, the number of input channels and hidden dimensions increase again, and the resolution of the image is further halved; the feature map of this stage contains the deepest information in the input image.

[0031] Furthermore, step S3 utilizes the environment-guided region representation learning module composed of a cross-attention mechanism and a feed-forward layer to learn the correlation between the fire region and the spatial environment data and the temporal environment data, and generates the representation A of the historical fire region by combining the spatial and temporal dimensions. The specific method is:

[0032] (1) Through the cross-attention mechanism, the historical fire area image sequence I s Features F s As Q, the spatial features F of multimodal data spatial As K and V, we can get the representation A of the learned fire area and spatial environment data spatial ;

[0033] (2) Through the cross-attention mechanism, the historical fire area image sequence I s Features F s As Q, the temporal features F of multimodal data temporal As K and V, we can get the representation A of the learning fire area and time environment data temporal ;

[0034] (3) Representation of fused spatial environment data spatial and the representation of time environment data A temporal , generating a representation A of the historical fire area.

[0035] Furthermore, in step S6, Binary Dice Loss and Binary Focal Loss are used as loss functions, and the neural network parameter θ is optimized by the gradient descent method; AUPRC, IOU and F1 scores are used as evaluation indicators to evaluate the prediction accuracy and robustness of the model.

[0036] The beneficial effects of the technical solution provided by the present invention are:

[0037] 1. Through multimodal data fusion, the present invention can comprehensively consider the impact of various environmental factors on the spread of fire, and achieve accurate prediction of the spread path of forest fires and accurate tracing of their origins.

[0038] 2. The present invention adopts technical means such as cross-attention mechanism and time series regression, and performs cross-domain feature extraction and fusion of multimodal data through deep learning, which significantly improves the model's fire prediction and backtracking capabilities in complex environments.

[0039] 3. During the model training process, the present invention combines simulated scene data of different fire starting points, fuel types and meteorological conditions to generate diversified training samples, which improves the model's adaptability to complex fire scenes and has strong practicality and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a forest fire spread and backtracking prediction method based on multimodal deep learning;

[0041] Figure 2 The accuracy comparison between the method of the present invention and the existing spatiotemporal data prediction method is as follows; DETAILED DESCRIPTION

[0042] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings and embodiments. Based on the technical solution of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0043] (I) Data collection and preprocessing: preparing a multimodal fire spread dataset for subsequent training and validation of forest fire spread and backtracking prediction models. The specific steps are as follows:

[0044] The study area was selected by using the Google Earth Engine platform to screen large fires with a fire area greater than 25,000 hectares in the GlobFire dataset as the study area.

[0045] Collect environmental data during the fire period in the study area, including terrain data, vegetation cover data, fuel distribution data, weather information, remote sensing satellite image sequences

[0046] Terrain data, vegetation cover data, and fuel distribution data are all collected from the LANDFIRE website: Terrain data refers to the Digital Elevation Model (DEM) of the study area, which reflects the undulating changes in the terrain. Terrain data is stored in raster format with a resolution of 30 meters; vegetation cover data refers to the vegetation cover information of the study area. Vegetation cover data is expressed in percentage form, ranging from 0% to 100%; fuel distribution data refers to the fuel distribution map in the study area. Fuel distribution maps usually include different types of fuels, such as grass, shrubs, trees, etc., and their corresponding fuel load values. Fuel distribution data is stored in raster format, and the value of each pixel represents the type of fuel and its density in the area.

[0047] Weather information is collected from the RemoteAutomatic Weather Station website, including key meteorological variables such as wind speed, wind direction, temperature, and humidity, with a time resolution of hourly level.

[0048] Remote sensing satellite image sequences: Remote sensing satellite image data from Sentinel-2 or Landsat series satellites covering the study area are obtained. These images have multispectral bands and are used to monitor changes in the fire area.

[0049] Fire area sequence generation.

[0050] Create a fire area sequence for training. Use the Farsite model to simulate fires based on the environmental information of the study area. Randomly select 100 fire points in each study area, and generate 100 simulated fire sequences. For each sequence, sample the historical fire area image sequence and the target fire area image sequence, respectively. s and I t The method is as follows: for a sequence, some fire area images I with the highest order of sampling are s , forming a historical fire area image sequence; sampling some fire area images with a forward arrangement order to form a target fire area image sequence I t .

[0051] Create a fire area sequence for verification. Based on the remote sensing satellite image sequence of the study area, the same method as above is used to sample the historical fire area image sequence and the target fire area image sequence, respectively. s ′ and I t 'express.

[0052] Data standardization and data labeling.

[0053] Normalization of terrain, vegetation cover, fuel distribution and weather data: Normalization of terrain, vegetation cover, fuel distribution and weather data is performed. Normalization is performed using the following formula:

[0054]

[0055] Among them, X is the original data value, μ is the mean of the data, σ is the standard deviation, and X′ is the standardized data. This step ensures that all input data have the same scale and distribution before model processing, avoiding deviations caused by different data scales during model training.

[0056] Remote sensing satellite image preprocessing: Atmospheric correction, geometric correction and radiation correction are performed on remote sensing satellite images to eliminate atmospheric and sensor effects. The corrected images are synthesized using multispectral technology and exported as RGB images.

[0057] Fire area annotation: The fire areas in the remote sensing satellite image sequence of the study area are annotated based on the fire data simulated by the Farsite model. The annotation results are expressed in the form of a binary fire mask, that is, each pixel value of 1 represents a fire area, and a value of 0 represents a non-fire area.

[0058] Create training and validation sets.

[0059] A training set based on simulated fire data and a validation set based on real fire data are prepared for model training and validation respectively. The organization of the training set and validation set is the same, the only difference is that the image sequence of the historical fire area is s , target fire area image sequence I t The sources are different. Historical fire area image sequence I s , multimodal temporal environment data X temporal (Where X temporal =X weather ), X weather The weather data of the simulated fire point in the study area and the multimodal spatial environmental data X spatial (Where X spatial ={X topography , X vegetation , X fuel}, X topography , X vegetation , X fuel The terrain of the simulated fire point in the study area, vegetation coverage, and fuel distribution data are used as the input of the model. The model predicts the fire area sequence at the target time. Therefore, the target fire area image sequence I t As supervision.

[0060] (ii) Using the Swin Transformer image encoder module and the sequence encoder module composed of the multi-head self-attention mechanism, the spatial features F are extracted from the input data respectively. spatial and time feature F temporal The specific steps are as follows:

[0061] (1) First, build a Swin Transformer image encoder module (SwinTransformerEncoder). This encoder is used to extract multi-scale spatial features of the input multimodal data. It consists of multiple stage modules (StageModule), each of which extracts multi-scale features of the input image at different resolutions. The Swin Transformer encoder is designed to extract meaningful features from the input multimodal data and provide high-quality feature representation for subsequent prediction tasks.

[0062] The entire encoder consists of three main stages. The first stage (Stage 1) receives the input image and converts its number of channels to hidden dimensions, while reducing the image resolution to one-fourth of the original size through downsampling operations. In this stage, the input image is processed by multiple SwinBlocks and convolutional layers to extract preliminary spatial features and output a feature map of size 64×64 pixels. The second stage (Stage 2) further processes the output of Stage 1. The number of input channels in this stage is increased, the hidden dimension is doubled, and the resolution of the image is halved again, generating a feature map of size 32×32 pixels. Through deeper SwinBlocks and convolutional layers, this stage extracts richer mid-scale spatial features. The third stage (Stage3) is the last stage of the encoder and is responsible for extracting the highest level of features. In this stage, the number of input channels and hidden dimensions are increased again, the resolution of the image is further halved, and the output is a feature map of size 16×16 pixels. The feature map of this stage contains the deepest level of information in the input image and is used for the final prediction of the model.

[0063] In each StageModule, the input data first passes through the Patch Merging layer, which is responsible for downsampling and increasing the number of channels. Subsequently, the data enters multiple SwinBlocks for processing. Each SwinBlock consists of a multi-head self-attention mechanism and a feedforward network. The attention mechanism is used to capture different features in space, and the feedforward network further processes these features. The SwinBlock in each StageModule works alternately in standard and displacement modes to capture richer feature information.

[0064] Through multi-stage feature extraction and downsampling operations, the Swin Transformer encoder can effectively extract multi-scale spatial features from the input data. These features will be used in subsequent complex tasks such as fire spread path prediction, providing powerful feature representation capabilities.

[0065] Extracting Historical Fire Area Image Sequences Using Swin Transformer Encoder I s Features F s The formula is as follows:

[0066] F s =Enc Swin (I s )

[0067] Among them, Enc Swin Represents the Swin Transformer encoder.

[0068] Extract spatial context data in image format using the Swin Transformer encoder X spatial and remote sensing satellite image sequence X satellite The features of the spatial feature F are concatenated spatial The formula is as follows:

[0069] F spatial =[Enc Swin (X spatial ), Enc Swin (X satellite )]

[0070] Among them, [·] represents the splicing process.

[0071] Next, a sequence encoder module (WxEncoder) composed of a multi-head self-attention mechanism is built to encode weather data. First, the input weather data is processed by the node feature extraction layer. This layer performs preliminary feature extraction on the data through a 1D convolution layer, and the size of the convolution kernel is set to 1 to ensure that only the channel dimension is transformed without affecting the time dimension. In this way, the input weather data is mapped to the hidden dimension (128 by default), and then the output of the convolution layer is nonlinearly transformed through the ELU activation function to generate richer node features.

[0072] Subsequently, the node features are fed into a multi-head self-attention mechanism. This mechanism consists of multiple attention heads that can compute dependencies between different time steps of a time series. In the final step, the output of the attention layer is processed by a linear layer to further map the features to the specified output dimension and then processed nonlinearly again by the ELU activation function.

[0073] The temporal context data X in sequence format is extracted using a sequence encoder module consisting of a multi-head self-attention mechanism. temporal and compare it with the remote sensing satellite image sequence X satellite The feature concatenation of temporal ,

[0074] F temporal =[Enc Wea (X temporal ), Enc swin (X satellite )]

[0075] Among them, Enc Wea Represents a sequence encoder consisting of a multi-head self-attention mechanism.

[0076] (III) Using the environment-guided region representation learning module composed of a cross-attention mechanism and a feed-forward layer, the correlation between the fire region and the spatial and temporal environment data is learned, and the representation A of the historical fire region is generated by combining the spatial and temporal dimensions. The specific steps are as follows:

[0077] (IV) The time series fire area regression module includes a self-attention layer and a feedforward layer. Based on the historical fire area representation A, it learns the time dependency of the target fire area and obtains the representation R of the target fire area to adapt to the time distribution interval of the target fire area. The specific steps are as follows:

[0078] Time series modeling, target fire area image sequence I t The timestamp encoding is used as Q, the representation A of the historical fire area is used as K and V, and the representation R of the target fire area is obtained through the cross attention mechanism.

[0079]

[0080] in,

[0081] Q=W q ·pos([p(1),...,p(T)]), K=W k A, V = W v A

[0082] Among them, pos([p(1),...,p(T)] represents the target fire area image sequence I t Timestamp encoding, which embeds the position of the image sequence into the query vector Q through the timestamp, and then associates it with the representation A of the historical fire area to capture the dependencies in the time series.

[0083] During the implementation, the input image features are first mapped to the query space through the convolutional layer. Then, the query, key, and value vectors are passed to the multi-head attention mechanism for interactive calculation. The attention mechanism calculates the correlation weights between the query and the key, and then applies these weights to the value vector to generate the final attention output, which is added element-by-element with the input image features and normalized by layers to ensure the stability of the model. Then, these features are further processed by the feedforward network and normalized again to finally obtain the representation R of the target fire area.

[0084] (V) Use the Swin Transformer image decoder module to perform spread prediction or backtracking prediction on the target fire area representation R and output the target fire area T. The specific steps are as follows:

[0085] (1) Build a Swin Transformer image decoder module (SwinTransformerDecoder), which is used to gradually restore the multi-scale features extracted by the encoder to a higher resolution and finally output the prediction results of the target fire area. The Swin Transformer decoder module consists of three upsampling stages. Each stage gradually increases the resolution of the feature map and finally outputs the prediction results of the target fire area. In the first stage of the decoder (Stage 4), the input is the feature map generated by the last stage of the encoder and the residual features of the middle layer of the encoder. This stage first increases the resolution of the feature map from 16×16 pixels to 32×32 pixels through an upsampling operation, while reducing the number of channels to half of the input. This process aims to gradually restore image details. The second stage (Stage 5) further processes the output feature map from Stage 4 and combines it with the residual features generated by the first stage of the encoder. In this stage, the resolution of the feature map is further increased to 64×64 pixels, and the number of channels is halved again. Through the processing of this stage, the decoder can gradually restore richer image details. The last stage (Stage 6) is the final upsampling stage of the decoder, which is responsible for restoring the output feature map of Stage 5 to the original high resolution (256×256) and reducing the number of channels to 1 to output the final fire area prediction result. This stage ensures that the spatial consistency and accuracy of the image are maintained at high resolution through fine upsampling operations.

[0086] The Swin Transformer decoder is used to perform spread prediction or backtracking prediction on the target fire area representation R, and the formula for outputting the target fire area T is as follows:

[0087] T=Dec Swin (R)

[0088] Among them, Dec Swin Represents the Swin Transformer decoder.

[0089] (VI) Train the deep neural network model on the virtual fire scene data generated by the simulator, and verify the model performance on the real fire scene data. The specific steps are as follows:

[0090] Binary Dice Loss and Binary Focal Loss are used as loss functions, and the neural network parameters θ are optimized by gradient descent method.

[0091] Binary Dice Loss calculation formula:

[0092]

[0093] Among them, p i represents the predicted value, g i represents the real value, and ∈ is a small constant to prevent division by zero.

[0094] Binary Focal Loss calculation formula:

[0095] FocalLoss = -α(1-p t ) γ log(p t )

[0096] Among them, p t is the probability predicted by the model, and α and γ are adjustment terms used to control the weights of easy-to-distinguish and difficult-to-distinguish samples.

[0097] AUPRC (Area Under Precision-Recall Curve), IOU (Intersection over Union) and F1 score are used as evaluation indicators to evaluate the prediction accuracy and robustness of the model.

[0098] The feasibility of the method of the present invention is verified by combining specific examples, as described below:

[0099] To verify the feasibility of the present invention, we conducted comparative experiments on the Sim2Real-Fire fire prediction dataset. The Sim2Real-Fire dataset consists of a large number of virtual simulated fire scenes and a small number of real fire scenes, covering a variety of terrains, vegetation, fuel types and meteorological conditions.

[0100] In the experiment, we compared the method of the present invention with the existing general spatiotemporal data prediction method, and the evaluation indicators included AUPRC, IOU and F1 score. The experimental results show that the performance of the present invention is significantly better than that of the existing methods in fire prediction and backtracking tasks under various complex environments. Specific experimental data and results are shown in the attached Figure 2 .

[0101] Through these experiments, the effectiveness and superiority of the method of the present invention can be verified. The present invention can effectively narrow the significant gap between simulated and real-world fire scenes, improve the prediction accuracy of fire spread paths and the ability to trace back the origin of fires, and provide important technical support for early warning and prevention of forest fires.

Claims

1. A forest fire spread and backtracking prediction method based on multimodal deep learning, comprising the following steps: S1, data collection and preprocessing: Collect multimodal environmental information of the fire period in the study area, including terrain, vegetation coverage, fuel distribution map, weather information and remote sensing satellite image sequences, standardize and annotate the data, and obtain the historical fire area image sequence I s , target fire area image sequence I t , Remote Sensing Satellite Image Sequence I satellite , multimodal temporal environment data X temporal , and multimodal spatial environment data X spatial , X spatial ={X topography , X vegetation , X fuel }); where the multimodal temporal environment data X temporal =X weather , X weather The weather data of the simulated fire point in the study area and the multimodal spatial environmental data X spatial ={X topography , X vegetation , X fuel }, X topography , X vegetation , X fuel The terrain, vegetation coverage, and fuel distribution data of the simulated fire point in the study area are used as the image sequence of the historical fire area. s , multimodal temporal environment data X temporal , and multimodal spatial environment data X spatial As the input of the model, the fire area sequence at the target time is predicted by the model, and the target fire area image sequence I t as supervision; S2, using the Swin Transformer image encoder module and the sequence encoder module composed of the multi-head self-attention mechanism, extracts spatial features F from the input data spatial and time feature F temporal ; S3, using the environment-guided region representation learning module composed of a cross-attention mechanism and a feed-forward layer, learns the correlation between the fire area and the spatial and temporal environment data, and generates a representation A of the historical fire area by combining the spatial and temporal dimensions; S4, the time series fire area regression module includes a self-attention layer and a feed-forward layer. Based on the historical fire area representation A, it learns the time dependency of the target fire area and obtains the representation R of the target fire area to adapt to the time distribution interval of the target fire area; S5, using the Swin Transformer image decoder module, performs spread prediction or backtracking prediction on the target fire area representation R, and outputs the target fire area T; S6, trains the deep neural network model on the fire scene data generated by the simulator, verifies the model performance on the real fire scene data, and adjusts the network parameters to reduce the gap between the simulated data and the real data.

2. The forest fire spread and backtracking prediction method based on multimodal deep learning according to claim 1 is characterized in that: Data collection and preprocessing of S1 include the following steps: S11, collect multimodal environmental information of the fire occurrence period in the study area, including topography, vegetation cover, fuel distribution map, weather information and remote sensing satellite image sequences; S12, generate fire area sequence: Create fire area sequences for training: Use the Farsite model to simulate fire according to the environmental information of the study area. For the study area, select the fire point, generate a simulated fire sequence for each fire point, and sample the historical fire area image sequence and the target fire area image sequence for each simulated fire sequence. s and I t express; Create a fire area sequence for verification; sample historical fire area image sequences and target fire area image sequences based on the remote sensing satellite image sequence of the study area; S13, standardize and label the data Standardize terrain, vegetation cover, fuel distribution, and weather data to avoid bias caused by different data scales during model training; Perform atmospheric correction, geometric correction and radiation correction on remote sensing satellite images to eliminate atmospheric and sensor effects; perform multi-spectral synthesis on the corrected images and export RGB images; Fire area labeling: Based on the fire data simulated by the Farsite model, the fire occurrence areas in the remote sensing satellite image sequence of the study area are labeled; the labeling results are expressed in the form of a binary fire mask, that is, each pixel value of 1 represents a fire area, and a value of 0 represents a non-fire area; S14, create training set and validation set The training set comes from the fire area sequences used for training that have been standardized and labeled; The validation set comes from the fire area sequences that have been standardized and annotated for validation; Historical fire area image sequence I s , multimodal temporal environment data X temporal (Where X temporal =X weather ), X weather The weather data of the simulated fire point in the study area and the multimodal spatial environmental data X spatial (Where X spatial ={X topography , X vegetation , X fuel }, X topography , X vegetation , X fuel The terrain of the simulated fire point in the study area, vegetation coverage, and fuel distribution data are used as the input of the model. The model predicts the fire area sequence at the target time. The target fire area image sequence I t As supervision.

3. The forest fire spread and backtracking prediction method based on multimodal deep learning according to claim 1 is characterized in that: The method of S2 is: S21, using Swin Transformer image encoder to extract historical fire area image sequence I s Features F s ; S22, use Swin Transformer image encoder to extract spatial environment data in image format X spatial and remote sensing satellite image sequence X satellite The features of the spatial feature F are concatenated spatial ; S23, using the sequence encoder module composed of a multi-head self-attention mechanism to extract the temporal environment data X in sequence format temporal and compare it with the remote sensing satellite image sequence X satellite The feature concatenation of temporal .

4. The forest fire spread and backtracking prediction method based on multimodal deep learning according to claim 3 is characterized in that: The method for constructing the Swin Transformer image encoder module is as follows: the Swin Transformer encoder is used to extract multi-scale spatial features of input multimodal data, including multiple stage modules StageModule, each stage module extracts multi-scale features of the input image at different resolutions; the Swin Transformer image encoder module includes three main stages; the first stage receives the input image and converts its number of channels into hidden dimensions, while reducing the resolution of the image to one-fourth of the original size through a downsampling operation, and the input image is processed by multiple SwinBlocks and convolutional layers to extract preliminary spatial features; The second stage further processes the output of the first stage. The number of input channels in this stage increases, the hidden dimension doubles, and the resolution of the image is halved again. Through deeper SwinBlock and convolutional layers, richer mid-scale spatial features are extracted. The third stage is used to extract the highest-level features. In this stage, the number of input channels and hidden dimensions increase again, and the resolution of the image is further halved. The feature map of this stage contains the deepest information in the input image.

5. The forest fire spread and backtracking prediction method based on multimodal deep learning according to claim 1 is characterized in that: Step S3 uses the environment-guided region representation learning module composed of a cross-attention mechanism and a feed-forward layer to learn the correlation between the fire region and the spatial environment data and the temporal environment data. The specific method of generating the representation A of the historical fire region by combining the spatial and temporal dimensions is as follows: (1) Through the cross-attention mechanism, the historical fire area image sequence I s Features F s As Q, the spatial features F of multimodal data spatial As K and V, we can get the representation A of the learned fire area and spatial environment data spatial ; (2) Through the cross-attention mechanism, the historical fire area image sequence I s Features F s As Q, the temporal features F of multimodal data temporal As K and V, we can get the representation A of the learning fire area and time environment data temporal ; (3) Representation of fused spatial environment data spatial and the representation of time environment data A temporal , generating a representation A of the historical fire area.

6. The forest fire spread and backtracking prediction method based on multimodal deep learning according to claim 1 is characterized in that: In step S6, Binary Dice Loss and Binary Focal Loss are used as loss functions to optimize the neural network parameters θ by gradient descent method; AUPRC, IOU and F1 scores are used as evaluation indicators to evaluate the prediction accuracy and robustness of the model.

Images