A satellite remote sensing fire point real-time monitoring method based on space-time feature fusion

Abstract

The application discloses a satellite remote sensing fire point real-time monitoring method based on space-time feature fusion, acquires historical remote sensing image data of geosynchronous satellite corresponding to a required prediction moment, constructs space information, band information and time information contained by each pixel point in each historical remote sensing image, and expands the number of fire points according to a copy method, combines a "rumination" method, completes the establishment of a data set, then takes the reconstructed pixel points of each historical remote sensing image in the data set as input, takes the actual fire point condition of each historical remote sensing image as output, trains a monitoring network model, finally constructs space information, band information and time information contained by each pixel point in a remote sensing image of a prediction moment of a required prediction date, inputs the reconstructed pixel points into the trained monitoring network model for feature extraction, and outputs final pixel classification results, generates a fire point map after visualization, and thus completes fire point monitoring.

Description

Technical Field

[0001] This invention relates to the technical field of target detection, and in particular to a real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion. Background Technology

[0002] Forest fires have occurred frequently in recent years. Forest fires are a crucial part of the natural ecosystem, offering various ecological benefits. Most naturally occurring fires do not require human intervention; however, in some cases, intense forest fires pose a serious threat to human life and property. Forest fires are characterized by their rapid spread. While early-stage fires are relatively small and easier to intervene in, they can quickly spread to surrounding areas, evolving into large, uncontrollable blazes. Therefore, rapid and accurate fire detection is essential. Remote sensing satellite imagery offers high resolution, allows for long-term, large-area monitoring, and is relatively inexpensive, making it an ideal tool for forest fire detection. Researchers have designed numerous fire detection algorithms based on different remote sensing satellites, and many countries have actively developed various fire-related products using these algorithms.

[0003] However, current fire detection algorithms primarily employ thresholding methods for fire point identification. These methods are highly sensitive to threshold settings. Since fires in different seasons and terrains exhibit different characteristics, the same threshold cannot be applied to other areas. Therefore, researchers have begun using deep learning algorithms with intelligent and automated features for fire point detection. One approach involves using semantic segmentation models to map fires on remote sensing images. However, these semantic segmentation models typically require high spatial resolution input images to provide detailed semantic information, and are generally applied to high-resolution sun-synchronous satellite images, making them difficult to use on geostationary satellites. Furthermore, sun-synchronous satellites have long revisit times, making near real-time monitoring difficult. Geostationary satellites, with their high temporal resolution, offer the possibility of real-time fire monitoring. Therefore, designing a deep learning method for fire point detection using geostationary satellite data is necessary. Summary of the Invention

[0004] This invention provides a real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion, which solves the problems of existing threshold methods for detecting fire points having fixed thresholds that cannot be changed according to actual conditions, and semantic segmentation models usually require input images with high spatial resolution, making it difficult to apply to geostationary satellite data.

[0005] This invention can be achieved through the following technical solutions:

[0006] A real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion is proposed. This method acquires historical geostationary satellite remote sensing image data corresponding to the time to be predicted, constructs the spatial, band, and temporal information contained in each pixel of each historical remote sensing image, expands the number of fire points using a replication method, and then combines this with a "rumination" method to complete the dataset construction.

[0007] Then, using the reconstructed pixels of each historical remote sensing image in the dataset as input and the actual fire situation of each historical remote sensing image as output, the SBT-FireNet monitoring network model is trained. Finally, the spatial, band, and temporal information contained in each pixel of the remote sensing image at the predicted time of the date to be predicted is constructed. The reconstructed pixels are then input one by one into the trained monitoring network model to extract features of spatial, band, and temporal information, and the classification results of each pixel are output. After visualization, a fire map is generated, thus completing the fire monitoring.

[0008] The spatial information is set as context spatial information centered on a pixel.

[0009] The band information is set as the value of each band at each pixel;

[0010] The time information is set as time series data composed of the band values ​​of pixels over D consecutive days.

[0011] Furthermore, the SBT-FireNet monitoring network model includes a feature extraction part and a classification part. The feature extraction part includes a spatial feature extraction module (SFE), a band feature extraction module (BFE), and a time series feature extraction module (TFE), which are used to extract the spatial environment information, band information, and time information corresponding to each pixel, respectively. The classification part includes a fully connected layer, which is used to comprehensively learn the features extracted by the feature extraction part and output the classification score corresponding to each pixel. If the score exceeds the threshold, it is determined to be a fire point; otherwise, it is determined to be a non-fire point.

[0012] Furthermore, a window of M × N size data centered on the pixel is selected as the input to the spatial feature extraction module SFE; the values ​​of each band on the pixel are combined into a column vector as the input to the band feature extraction module BFE; and the values ​​of each band of the pixel over D consecutive days are combined into multiple column vectors as the input to the time series feature extraction module TFE.

[0013] 1. Further, the spatial feature extraction module SFE is configured as a Vision transform encoder structure, the band feature extraction module BFE is configured as a multilayer perceptron (MLP), and the time series feature extraction module TFE includes a multilayer perceptron (MLP) and a Bi-LSTM network. Band data from consecutive days D are input into the multilayer perceptron (MLP) to learn the band features of the pixel on each day. Then, the learned band features from D days are input into the Bi-LSTM network to learn the temporal features of the pixel on D days.

[0014] Furthermore, the multilayer perceptron (MLP) comprises an input layer for receiving input information, two hidden layers for learning implicit features in the input information, and an output layer for outputting band features. The number of neurons in the hidden layers are 1024 and 512, respectively, and the number of neurons in the output layer is 128.

[0015] Furthermore, by using cloud, water, and underlying surface masks to copy fire points into cloudless and waterless forest / grassland pixels, the number of fire points in each historical remote sensing image is expanded. Then, at a 1:1 ratio, pixels corresponding to fire points and non-fire points are randomly selected from each historical remote sensing image to form an initial training set. The selected pixels are then input into the monitoring network model for initial training using a "rumination" method. The pre-trained monitoring network model is then used to detect each historical remote sensing image. A certain number of misjudged pixels are then extracted and added to the initial training set, and re-input into the monitoring network model for training, thus obtaining a trained monitoring network model.

[0016] Furthermore, for each historical remote sensing image, the number of newly added misjudged pixels does not exceed 100.

[0017] The beneficial technical effects of this invention are as follows:

[0018] 1) The SBT-FireNet monitoring network model can automatically extract the spatial, band, and temporal features of fire points without the need for manual design of additional features. The extracted features are comprehensively classified, and the pixels are finally classified as fire points / non-fire points, thus accurately extracting fire points on geostationary satellite data. It has good robustness in areas such as clouds, cloud edges, and thin clouds.

[0019] 2) When constructing the initial training dataset using the replication method, the number of fire points should be expanded as much as possible. After using the "rumination" method, the final training dataset can be constructed, so that the constructed dataset includes as many types of pixels as possible. This allows the SBT-FireNet monitoring network model to fully learn the features of fire points / non-fire points and enhance the model's robustness to clouds and cloud edges.

[0020] 3) Based on the experimental equipment of this invention, the SBT-FireNet monitoring network model is used to detect fire points in a 400*300 conventional remote sensing image. The time required is no more than 10 seconds, which is much lower than the 10-minute time resolution of the Himawari-8 satellite, and can fully realize real-time monitoring of fire points. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0022] Figure 2 This is a schematic diagram of the monitoring network model of the present invention;

[0023] Figure 3 This is a schematic diagram of the ViT structure of the present invention;

[0024] Figure 4 This is a schematic diagram of the BFE structure of the present invention;

[0025] Figure 5 This is a schematic diagram of the TFE structure of the present invention;

[0026] Figure 6 This is a schematic diagram of the input data for the monitoring network model of the present invention. Detailed Implementation

[0027] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments.

[0028] like Figure 1As shown, this invention provides a real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion. It acquires historical geostationary satellite remote sensing image data corresponding to the time to be predicted, constructs spatial, band, and temporal information for each pixel in each historical remote sensing image, expands the number of fire points using a replication method, and then completes the dataset construction using a "rumination" method. The SBT-FireNet monitoring network model is then trained using the reconstructed pixels of each historical remote sensing image in the dataset as input and the actual fire point situation of each historical remote sensing image as output. Finally, it constructs the spatial, band, and temporal information for each pixel in the remote sensing image of the predicted date, inputs the reconstructed pixels one by one into the trained monitoring network model to extract features from the spatial, band, and temporal information, and outputs the classification results for each pixel. After visualization, a fire point map is generated, thus completing fire point monitoring. The spatial information is set as the contextual spatial information centered on the pixel; the band information is set as the value of each band on the pixel; and the temporal information is set as the time series data composed of the band values ​​of the pixel over D consecutive days. This approach, utilizing the SBT-FireNet monitoring network model for fire detection, automatically extracts relevant fire features, avoiding the limitation of fixed thresholds in traditional thresholding methods for fire detection. This improves the accuracy of fire detection. Furthermore, by integrating spatial, spectral, and temporal features of the fire point into the fire detection calculation, it avoids data gaps and provides a solid data foundation for the accuracy of subsequent detections. Specifically:

[0029] Step 1: Dataset Creation Strategy

[0030] Considering the key characteristics of fire points, the input data should contain as much information as possible about the band, spatial, and temporal aspects of each pixel. The input data mainly consists of the following three parts: context patch, band data, and time series data.

[0031] like Figure 6 As shown, a pixel's context patch refers to an M × N window centered on that pixel. The context patch contains the pixel's environmental information, and the context block contains the spatial information of the central pixel. A pixel's band data refers to the value of each band on that pixel. Band data in a continuous time series refers to the sequence data composed of the band values ​​of a pixel within a certain continuous time period. Continuous time refers to the same moment over D consecutive days.

[0032] Augmenting data using data augmentation techniques:

[0033] By using cloud, water, and underlying surface masks, fire points are replicated into cloudless, waterless forest / grassland pixels. This increases the number of fire points while maintaining their plausibility. Furthermore, because the fire points are replicated in a new environment, they also enrich the spatial information in the training dataset. This novel data augmentation method can increase the number of fire points by up to 20 times. Then, an equal number of non-fire points are randomly selected to form the initial training dataset. In the initial training dataset, the ratio of fire points to non-fire points is 1:1.

[0034] Constructing the final dataset: First, train an imperfect monitoring network model using the initial training dataset. Then, use the imperfect monitoring network model to predict all images in the training set (excluding images in the test dataset). Extract misclassified pixels from each image and add them to the initial training dataset. Set an upper limit on the number of misclassified pixels, such as a maximum of 100 misclassified pixels per image. These can be added to the initial training dataset. This method is called the "rumination" method.

[0035] Therefore, following the above method, we only need to obtain the historical remote sensing image data of geostationary satellites corresponding to the time when the prediction is needed, construct the spatial information, band information and temporal information contained in each pixel of each historical remote sensing image, establish a dataset, and then use the reconstructed pixels of each historical remote sensing image in the dataset as input and the actual fire point situation of each historical remote sensing image as output to train the monitoring network model. During training, 40% of the training set is not used for training and is used as a validation set to observe the training status of the network in real time.

[0036] Step 2: Construct the SBT-FireNet monitoring network model structure

[0037] The model proposed in this invention is named SBT-FireNet, or SBT for short, and its structure is as follows: Figure 2As shown, this is a customized network designed based on the characteristics of fire points. First, the fire point detection problem is transformed into a classification problem, that is, classifying each pixel in the image as a fire point or a non-fire point. The proposed monitoring network model mainly consists of two parts: a feature extraction part containing three modules and a classification part that outputs classification scores. The feature extraction part includes three modules: Spatial Feature Extraction (SFE), Band Feature Extraction (BFE), and Temporal Feature Extraction (TFE), used to extract environmental spatial information, band information, and temporal information of the fire point, respectively. The classification part consists of a fully connected layer with 674 nodes, which can comprehensively learn the features extracted by the feature extraction part, classifying pixels with high fire classification scores as fire points and pixels with low fire classification scores as non-fire points. Based on this combination of strategies, this method can overcome the problems of applying deep learning to geostationary satellites and fully consider the characteristics of fire points.

[0038] 1. SFE module

[0039] The input to SFE is a context patch, and its function is to learn the spatial features of points from the context patch. Its main component is ViT. ViT is a successful application of the transformer structure in computer vision, directly applying the transformer to image classification tasks and achieving accurate and precise results. The ViT structure is as follows: Figure 3 As shown.

[0040] In the field of natural language processing, the transformer was first proposed for text translation tasks. Transformers are typically used to process one-dimensional (1D) data. To apply the transformer to two-dimensional (2D) image data, ViT... It was reshaped into a series of two-dimensional image patches. Then the two-dimensional image block is flattened into a one-dimensional block. H, W These are the height and width of the two-dimensional image, respectively. p The width and height of the two-dimensional image block. n The number of two-dimensional image patches, c This represents the number of channels (bands) in the 2D image. To represent the classification information of the 2D image, a learnable embedding called a class token is applied to the one-dimensional block. Furthermore, to preserve positional information, positional embeddings are added to the flattened one-dimensional block.

[0041] The Transformer encoder mainly consists of a multi-head self-attention layer, a normalization layer, and an MLP. The class token of the Transformer encoder is input into the MLP head, and finally, the MLP head outputs a score for the 2D image in each class. To enable ViT to perform this task, p Setting it to 7 divides the input of the SFE into 9 one-dimensional blocks. After adding the same learnable feature tokens as the class labels to the flattened 9 blocks, they are fed into the transformer encoder. The feature token output of the transformer encoder is used as the output of the SFE. The feature tokens are the extracted spatial features.

[0042] 2. BFE module

[0043] BFE structure as follows Figure 4 As shown. If each pixel has 6 acquisition bands, then the input to the BFE is a column vector of shape (1,6), which includes the value of each band of the pixel. In the BFE, a multilayer sensor machine (MLP) is used to map the 6-dimensional band features of the input to a higher dimension to learn the features implied in the bands. Specifically, the MLP consists of an input layer for receiving input information, two hidden layers for learning the implicit features in the input information, and an output layer for outputting the learned features. The number of neurons in the two hidden layers are 1024 and 512, respectively, and the number of neurons in the output layer is 128.

[0044] 3. TFE module

[0045] TFE structure as Figure 5 As shown, the TFE consists of a multilayer inductively coupled plasma (MLP) and a Bi-LSTM. The input to the TFE is the time-series data of the pixel, with a shape of (D, 6), where D represents the length of the time series. Assuming D=7, and each pixel has 6 acquisition bands, then the band data for one day is a one-dimensional column vector with a data volume of 6, and the band data for 7 consecutive days is a 7*6 matrix structure. In the TFE, the band data of D days is first input into the MLP to learn the band features of the pixel for each day. Then, the learned band features of D days are connected to the Bi-LSTM to learn the temporal features of the pixel over D days.

[0046] Bi-LSTM extracts features from time-series data in two ways. Once it learns the data information from the previous time point, the LSTM structure transfers the learned features to the next LSTM structure. The next LSTM structure can then fully learn the changing features contained in the time-series data. These changing features can reflect the changes in sample band information in the continuous time series, thus providing a basis for distinguishing whether a pixel is a fire point.

[0047] LSTM was initially used to solve the vanishing and exploding gradient problems during training on sequence data, involving the following three steps:

[0048] 1) It receives and learns the features (ct-1) transmitted from the previous LSTM structure (previous node), and selectively forgets some content. In this process, the LSTM selectively learns the feature output of the previous node, "forgets" the relatively unimportant features, and "remembers" the relatively important features.

[0049] 2) Learn the features of a node by learning the input of the node (ct) and “memorizing” the important features contained in the node.

[0050] 3) Combine the inputs from the previous node and the current node to perform integrated learning, output the learning results, and pass the output to the next node.

[0051] Through the steps described above, LTSM can learn the input of the current node while retaining important features learned by the previous node. For LTSM, Bi-LSTM not only undergoes a process of passing information from the previous node to the next, but also from the next node to the previous, ensuring that it can learn information from the given context more completely. Therefore, Bi-LSTM was chosen to learn the temporal features of fire points.

[0052] Step 3: Real-time monitoring

[0053] As described above, spatial, band, and temporal information is constructed for each pixel in the remote sensing image of the predicted date and time. The reconstructed pixels are then input into the trained monitoring network model one by one. The three feature extraction modules in the monitoring network model are used to extract features from the spatial, band, and temporal information respectively. The extracted features are then uniformly input into a fully connected layer for fusion and classification. Finally, the classification results of each pixel are output. After visualization, a fire map is generated, thus completing the fire monitoring.

[0054] To verify the feasibility of the real-time monitoring method of this invention, remote sensing image data acquired by the Himawari-8 satellite were obtained. The climate of the study area is subtropical or tropical monsoon, with a long-term annual temperature range of 12-29℃.

[0055] As the successor to the MTSAT series of geostationary meteorological satellites, Himawari-8 was launched on July 7, 2015, and is equipped with the Advanced Himawari Imager (AHIs). AHIs has 16 observation bands: 10 infrared bands, 3 near-infrared bands, and 3 visible light bands. The spatial resolutions for the infrared, near-infrared, and visible light bands are 0.5 km–1 km, 1 km–2 km, and 1 km–2 km, respectively, with a total observation interval of 10 minutes.

[0056] The bands used in this method are listed in Table 1.

[0057]

[0058] All code in this method was written in Python 3.6. For building the deep learning models, PyTorch version 1.2 was used. All experiments were conducted on an Intel Core i9-10900K CPU @ 3.70 GHz, 128 GB RAM, and an NVIDIA GeForce GTX 3080. During the training of each deep learning model, the Adam optimizer was used as the parameter optimizer, the cross-entropy loss function was used, the epoch was set to 500, the batch size was set to 100, and the learning rate was set to 10⁻⁶.

[0059] In addition, accuracy, precision, recall, F1-score (F1), false negative rate (MD), and false error rate (ED) are used as key performance indicators of the model, and their calculation formulas are as follows:

[0060]

[0061] TP indicates a true positive result; TN indicates a true negative result; FP indicates a false positive result; and FN indicates a false negative result.

[0062] The SBT-fireNet monitoring network model (SBT) was compared with common deep learning classification models such as ResNet50 (Res50) and ViT. ResNet50 is a classic convolutional neural network classification model that uses multi-layer convolutional pooling structures to progressively learn the semantic information of images and uses residual structures to prevent gradient vanishing or gradient exploding. ViT is a successful application of transformer in image classification. It converts image data into blocks and then compresses them. It uses multi-layer attention mechanisms to iteratively learn the features of each block and finally outputs the classification of the head image.

[0063] Table 2 shows that SBT has the best overall performance. Furthermore, Res50 identifies all pixels as non-fire pixels, getting stuck in a local optimum. Therefore, the classification standard of Res50 was appropriately lowered, including fire pixels with lower confidence levels; this lower-standard Res50 is called Res50-low. Res50-low has very low accuracy and identifies a large number of pixels as fire pixels. Res50's MD value reaches 1.0, while Res50-low's ED value reaches 0.999, indicating that neither model is suitable for fire detection tasks. SBT has an accuracy of 0.747, a recall of 0.761, and an F1 score of 0.754. Compared to ViT, SBT improves accuracy and recall by 0.352 and 0.665, respectively. SBT has the highest F1 score, indicating that SBT is more suitable for fire detection. Since ViT and Res50 were not designed for geostationary satellite-based fire detection tasks and failed to comprehensively utilize all aspects of fire information, this result is expected.

[0064]

[0065] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion, characterized in that: To obtain historical geostationary satellite remote sensing imagery data corresponding to the time of prediction, spatial, band, and temporal information of each pixel in each historical remote sensing image is constructed. The number of fire points is expanded using a replication method, and then combined with a "rumination" method to complete the dataset construction. Then, using the reconstructed pixels of each historical remote sensing image in the dataset as input and the actual fire situation of each historical remote sensing image as output, the SBT-FireNet monitoring network model is trained. Finally, the spatial, band, and temporal information contained in each pixel of the remote sensing image at the predicted time of the date to be predicted is constructed. The reconstructed pixels are then input one by one into the trained monitoring network model to extract features of spatial, band, and temporal information, and the classification results of each pixel are output. After visualization, a fire map is generated, thus completing the fire monitoring. The spatial information is set as context spatial information centered on a pixel. The band information is set as the value of each band at each pixel; The time information is set as time series data consisting of the band values ​​of pixels over D consecutive days; The SBT-FireNet monitoring network model includes a feature extraction part and a classification part. The feature extraction part includes a spatial feature extraction module (SFE), a band feature extraction module (BFE), and a time series feature extraction module (TFE), which are used to extract the spatial environment information, band information, and time information corresponding to each pixel, respectively. The classification part includes a fully connected layer, which is used to comprehensively learn the features extracted by the feature extraction part and output the classification score corresponding to each pixel. If the score exceeds the threshold, it is determined to be a fire point; otherwise, it is determined to be a non-fire point. The spatial feature extraction module SFE is configured with a Vision transform encoder structure, the band feature extraction module BFE is configured with a multilayer perceptron (MLP), and the time series feature extraction module TFE includes a multilayer perceptron (MLP) and a Bi-LSTM network. Band data from consecutive days D are input into the multilayer perceptron (MLP) to learn the band features of the pixel on each day. Then, the learned band features from D days are input into the Bi-LSTM network to learn the temporal features of the pixel on D days.

2. The real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion according to claim 1, characterized in that: A window of M × N size is selected centered on the pixel as the input to the spatial feature extraction module SFE; the values ​​of each band on the pixel are combined into a column vector as the input to the band feature extraction module BFE; and the values ​​of each band of the pixel over D consecutive days are combined into multiple column vectors as the input to the time series feature extraction module TFE.

3. The real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion according to claim 1, characterized in that: The multilayer perceptron (MLP) consists of an input layer for receiving input information, two hidden layers for learning implicit features in the input information, and an output layer for outputting band features. The number of neurons in the hidden layers are 1024 and 512, respectively, and the number of neurons in the output layer is 128.

4. The real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion according to claim 1, characterized in that: By using cloud, water, and underlying surface masks to copy fire points into cloudless and waterless forest / grassland pixels, the number of fire points in each historical remote sensing image is expanded. Then, at a 1:1 ratio, pixels corresponding to fire points and non-fire points are randomly selected from each historical remote sensing image to form an initial training set. The selected pixels are then input into the monitoring network model for initial training using a "rumination" method. The pre-trained monitoring network model is then used to detect each historical remote sensing image. A certain number of misjudged pixels are then extracted and added to the initial training set, and re-input into the monitoring network model for training, thus obtaining a trained monitoring network model.

5. The real-time monitoring method for satellite remote sensing fire points based on spatiotemporal feature fusion according to claim 4, characterized in that: For each historical remote sensing image, the number of newly added misclassified pixels shall not exceed 100.

Images