A fire prediction method integrating multi-source time series data suitable for edge computing

Abstract

The present invention discloses a fire prediction method for fusing multi-source time series data suitable for edge computing, including collecting multi-view RGB images, multi-view infrared images and meteorological data, and preprocessing the data; constructing a Smokey Net prediction model with CNN as the backbone network, inputting the preprocessed data into the prediction model, and performing multi-source data fusion; selecting three attention mechanisms: sparse attention mechanism, strong attention mechanism and multi-head attention mechanism; adding an external register module and a long short-term memory network; and training the Smokey Net prediction model. The present invention solves the problem of insufficient sample data when predicting wild fires, and improves the accuracy of long-term monitoring. Thereby helping to reduce casualties and property losses caused by fires, and improving the efficiency and timeliness of responding to fires.

Description

Technical Field

[0001] The present invention belongs to the field of fire detection, and in particular relates to a fire prediction method that integrates multi-source time series data and is suitable for edge computing. Background Art

[0002] The concept of wildfire prediction refers to the use of data analysis, machine learning, and deep learning technologies to establish prediction models to identify potential wildfire risks through the analysis of historical data and specific features to predict wildfires so that preventive measures can be taken in advance or in real time. For example, a wildfire prediction system suitable for edge computing can use sensor data and weather data to predict the possibility of wildfires. The system first collects historical sensor data, including information such as temperature, humidity, smoke density, and weather data such as wind speed and air pressure. Then, this data is used to train neural network models such as LSTM to learn patterns and features in the data. Ultimately, through the wildfire prediction system, potential fire risks can be identified in advance to help reduce the losses and harm caused by wildfires.

[0003] At present, an important problem in wildfire prediction is that wildfire prediction requires a large amount of data support, including sensor data, meteorological data, etc. However, there are problems with data quality, which leads to insufficient data samples and ultimately affects the accuracy of the prediction model. There are currently three key engineering problems:

[0004] (1) Wildfires generally last a long time. It takes dozens of hours from the generation of sparks to the formation of obvious smoke and open flames. Obviously, it is not suitable to use the equal-interval sampling method to detect them. How to analyze a large amount of data under the premise of long-term monitoring is a major engineering problem.

[0005] (2) Problems of insufficient detection perspective and data samples. In wildfire detection, fixed detection perspective and insufficient data samples are important issues, especially for rare but extremely important events such as wildfires. This will lead to insufficient model training and affect the model's generalization ability and accuracy.

[0006] (3) At night, the brightness of fireflies can be easily misreported as dark fires. Frequent false alarms may reduce the reliability of fire detection models, making it difficult to identify real signs of fire, thereby affecting the efficiency of emergency response. This is also a major engineering problem.

[0007] For these three problems, existing research on wildfire prediction solutions focuses on continuously improving the identification of wildfires, but does not consider the above two engineering problems. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a fire prediction method that integrates multi-source time series data and is suitable for edge computing. In the prediction of wild fires, the problem of insufficient sample data is solved and the accuracy of long-term monitoring is improved. This helps reduce casualties and property losses caused by fires and improves the efficiency and timeliness of responding to fires.

[0009] Technical solution: A fire prediction method for fusing multi-source time series data suitable for edge computing of the present invention is characterized by comprising the following steps:

[0010] Step 1: Collect multi-view RGB images, multi-view infrared images and meteorological data of the target area, and pre-process the multi-view RGB images and meteorological data;

[0011] Step 2, performing Gaussian smoothing on the multi-view infrared images;

[0012] Step 3: Use CNN as the backbone network to build a Smokey Net prediction model, input the preprocessed multi-view RGB images and meteorological data, and the multi-view infrared images after Gaussian smoothing into the Smokey Net prediction model to perform multi-source data fusion;

[0013] Step 4: Select three attention mechanisms: sparse attention mechanism, strong attention mechanism, and multi-head attention mechanism;

[0014] Step 5: Add external register module and long short-term memory network;

[0015] Step 6: Train the Smokey Net prediction model, update the model weights through stochastic gradient descent, and adjust the loss function and optimizer to obtain the trained Smokey Net prediction model. Use the trained Smokey Net prediction model to predict fires in the target area.

[0016] Furthermore, step 1 specifically includes the following steps:

[0017] Step 1.1: Select data from different sources, including RGB images of wildfires at different angles taken by cameras in different directions, infrared images, and relevant meteorological data on temperature and humidity. The original data set is represented as follows:

[0018] in represents the original data set, express time Red picture from perspective, express time Green picture from perspective, express time Blue picture from perspective, express time The infrared image corresponding to the RGB image of the viewing angle, express Temperature data at the moment, express Wind data at the moment, express Humidity data at all times;

[0019] Step 1.2: Preprocess the temperature, humidity, and wind data in the data set to make their range consistent with the image pixel range of 0-255 to reduce the weight bias problem in the model training process. The preprocessing method for temperature, wind, and humidity data is normalization, which is expressed by the following formula:

[0020] ;

[0021] The preprocessed data set is expressed as:

[0022] in for Normalization results of temperature data at each moment; for Normalization result of wind data at that time; for Normalization results of humidity data at all times; express Temperature data at the moment, express Wind data at the moment, express Humidity data at all times; is the minimum temperature, is the maximum temperature, is the minimum wind speed, is the maximum wind speed, is the minimum humidity value, is the maximum humidity;

[0023] Step 1.3: Label or annotate the data, that is, assign the correct label or category to each sample, and use computer vision technology to automatically label, that is, use object detection and semantic segmentation technology to detect the flame or smoke area in the image;

[0024] Step 1.4: Divide the dataset into training set, validation set and test set.

[0025] Furthermore, step 2 specifically includes the following steps:

[0026] Step 2.1: Use Gaussian filtering to weight the neighborhood around each data point so that the farther the data point is from the center point, the smaller the weight. Set a size of Gaussian kernel , the center point is , for infrared images Each pixel in is denoted as , the following formula is expressed:

[0027] ; ;

[0028] in, and is the index of the Gaussian kernel, and the range of the loop traversal is , is the pixel in the input image The gray value of is the pixel in the output image after convolution The value of is the standard deviation of the Gaussian kernel, is the base of natural logarithms;

[0029] Step 2.2, determine the window size. Considering the characteristics of the image, the noise level, and the application requirements, a 7×7 window is selected to better smooth the image.

[0030] Furthermore, step 3 specifically includes the following steps:

[0031] Step 3.1, use CNN to extract each frame of RGB three-color image and infrared image respectively, select the last layer of the penultimate convolution block of VGG16, that is, block5_conv3 layer, for feature extraction. The workflow of VGG16 is input time Red picture from perspective , time Green picture from perspective , time Blue picture from perspective and infrared images The feature maps obtained after VGG16 are recorded as , the following formula is expressed:

[0032] ;

[0033] Step 3.2: Perform multi-source data fusion, and fuse the feature images of each RGB three-color image and infrared image extracted by CNN with the meteorological factors of temperature, humidity and wind to generate a new feature image. First, assume that the three data sources of temperature, humidity and wind direction are The weights of the moments are , then the formula of weighted fusion is expressed as:

[0034] ;

[0035] After the fusion is completed The characteristic of the moment is recorded as , the formula is:

[0036] .

[0037] Furthermore, step 4 is as follows: The principle of the sparse attention mechanism is as follows: select the feature pixels of the RGB three-color time-series image and the infrared time-series image, and take the first pixel of the red R in the RGB three-color time-series image as the first pixel. For example, is the first Moment elements, remember for F 1 , so the Elements The time series is the input sequence represented by , is the length of the sequence. First, the similarity is calculated. For each position Calculate it with other positions The correlation between them is measured by inner product, Euclidean distance or cosine similarity. This process is a black box calculation inside the attention mechanism. According to the calculated similarity, the The most relevant positions form a sparse subset The method used is to select according to the similarity threshold, for each position , using the selected sparse subset Position in , the attention weight is calculated by the softmax function ,in The formula is as follows:

[0038] ;

[0039] in Indicates location and The similarity between is the temperature parameter used to control the smoothness of the distribution,

[0040] Finally, according to the obtained attention weight , get the location Output characteristics , the formula is:

[0041] ;

[0042] The principle of the hard attention mechanism is as follows: in the temperature data, select the first moments, of which is the first Moment elements, remember for , then its input sequence , is the length of the sequence, at each time step , Hard Attention will choose a position according to the greedy strategy , as the focus of attention at the current time step, and then according to the selected position , calculate the attention weight vector of the current time step , where only the position The weight of the position is 1, the weight of other positions is 0, and the attention weight vector is used Perform weighted summation on the value vector to obtain the output vector for the current time step , the formula is as follows:

[0043] ;

[0044] Finally, the output vector sequence is obtained based on the position selected at each time step ;

[0045] The principle of the Multi-Head Attention mechanism is as follows: select the feature pixels of the RGB three-color image and the infrared image and the first moments, of which is the first Moment elements, remember for , then its input sequence is:

[0046] ;

[0047] is the length of the sequence. First, we pass the input sequence Multiply by the query matrix , key matrix Sum Matrix , get the query ,key Sum , then split the query, key, and value into Head, get ,in Indicates For each head , calculate the attention score , the formula is as follows:

[0048] ;

[0049] in, is the dimension of the key;

[0050] Compute the attention output of each head , the formula is as follows:

[0051]

[0052] Finally, the attention outputs of all heads are concatenated and multiplied by the output matrix The final output is shown in the following formula:

[0053] .

[0054] Furthermore, step 5 specifically includes the following steps:

[0055] Step 5.1, add external registers, use external registers to store additional feature images and continuously update to store feature images, thereby enhancing its performance and generalization ability. The working process is to selectively store the current feature images in the external register during training, replace the old feature images, and update 1 group each time to ensure that there are always 10 groups of data in the external register; finally, in the forward propagation process of the model, combine the current input data and the feature images in the external register;

[0056] Step 5.2: Introduce the LSTM framework, where The calculation formula of the LSTM state at a certain moment is:

[0057] ;

[0058] Where: is the input vector; For the forgotten door at the moment The activation vector of For the input gate The activation vector at the moment; For the output gate The activation vector at the moment; for The memory cell activation vector at the moment; for The hidden state activation vector at time instant; is the input kernel of each gate; is the loop kernel of each gate; for deviation; is the Sigmoid activation function; tanh is the hyperbolic tangent activation function.

[0059] Furthermore, in step 6, the model weights are updated by stochastic gradient descent as follows: the learning rate is determined, the loss function and the optimizer settings are continuously adjusted, and the learning rate after the update is As shown below

[0060]

[0061] In the above formula, the initial momentum power is set to 0.8 and the initial learning rate is set to is 0.001, and uses represents the maximum number of iterations, Indicates the current iteration number.

[0062] The present invention also discloses a fire prediction system for fusing multi-source time series data suitable for edge computing, including a multi-view multi-directional data module, an infrared data smoothing processing module, a new Smokey Net architecture module and a training parameter setting module;

[0063] The multi-view and multi-orientation data module includes RGB images and infrared images of different angles before and after a wildfire, as well as temperature, humidity, and wind meteorological data taken by cameras in different orientations; this module is used to provide more comprehensive and diverse information, improve the robustness of the system, and make it better adaptable to different environmental conditions and fire characteristics;

[0064] The infrared data smoothing processing module is used to remove the nighttime interfering light sources contained in the infrared image;

[0065] The new Smokey Net architecture module combines CNN, LSTM and multi-modal sparse transformer. The SmokeyNet architecture can fully combine the input RGB images, infrared images, temperature, humidity, wind and meteorological data to fully utilize the information of different types of data, accurately detect wildfire smoke and predict the development trend of wildfires;

[0066] The training parameter setting module adjusts relevant parameters to improve the training efficiency, generalization ability and performance of the field fire detection model.

[0067] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0068] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: First, the present invention improves data utilization efficiency. This is mainly reflected in the fact that the present invention uses a sparse attention mechanism to effectively solve the problem of too much sample data in wild fire prediction. Traditional methods usually require a large amount of labeled data to train the model, while the sparse attention mechanism can use a small number of samples for efficient learning while maintaining accuracy. This method greatly improves data utilization efficiency. Secondly, the present invention introduces an external register to store additional feature images and continuously updates them to store the best feature images, thereby enhancing its performance and generalization ability and improving the accuracy of long-term monitoring. The introduction of the new Smokey Net architecture model enables the present invention to quickly respond to potential fire risks in fire prediction and monitoring. Compared with traditional methods, it can not only detect potential fire signs earlier, but also take more timely and effective measures before the fire occurs, such as quickly dispatching resources, warning the community and preparing emergency response plans, which will effectively reduce casualties and property losses caused by fire. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is the overall idea framework diagram of the present invention;

[0070] Figure 2 is a flowchart of constructing a data set of the present invention;

[0071] Figure 3 The framework diagram of the Smokey Net architecture model constructed in the present invention;

[0072] Figure 4 Schematic diagram of the principle of feature extraction for VGG16;

[0073] Figure 5 This is a diagram of the multi-head sparse attention mechanism model;

[0074] Figure 6 It is a working diagram of the external register;

[0075] Figure 7 This is a schematic diagram of the principle of LSTM. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0077] Based on the existing artificial intelligence and wildfire prediction technology, the present invention proposes solutions to the three engineering problems faced by the existing wildfire prediction algorithms: insufficient data samples, how to analyze data under the premise of long-term monitoring, and firefly interference. It can solve the above problems well and improve the accuracy of wildfire prediction.

[0078] The optical camera image and infrared image detection algorithm of the present invention, which combines multiple view angles and adopts deep learning, includes a multi-view multi-directional data module; an infrared data smoothing processing module; a new Smokey Net architecture module; and a training parameter setting module. The overall process is as follows Figure 1 shown.

[0079] Among them, the multi-view and multi-directional data module, the infrared data smoothing processing module, and the new Smokey Net architecture module are the main innovations of the present invention, which effectively improve the accuracy of the overall prediction of wild fires.

[0080] Ideal wildfire monitoring should have high-precision fire prediction capabilities, be able to accurately predict the time, location and scale of fires, and have high temporal and spatial resolutions, be able to make short-term and long-term fire predictions, and be able to make refined fire predictions for different regions;

[0081] The use of multi-view and multi-directional wildfire data modules is of great significance for fire monitoring and prevention. By collecting multi-view data, including infrared images, satellite images, and meteorological data, it can fully understand the development of fires and improve the accuracy of fire detection and prediction. At the same time, the use of multi-view data can provide rich information to help identify fire hotspots, fire spread, and possible fire causes, providing more comprehensive support for fire monitoring and emergency response. In the wild environment, infrared sensors may be disturbed by fireflies at night. Therefore, the infrared smoothing module can help improve the stability and accuracy of infrared data, thereby improving the performance of the fire detection system. The new Smokey Net architecture suitable for edge computing constructed by the present invention is an innovative spatiotemporal grid image classification method for wildfire smoke detection. The architecture integrates CNN, LSTM, and multimodal sparse transformers, and receives RGB and infrared video sequences and related meteorological data as input. CNN is responsible for extracting pixel representations, and processing time series data through multimodal sparse attention and LSTM to predict the development trend of fires. At the same time, an external storage device is introduced to store historical fusion data, which is input into LSTM together with the processed data to further improve the efficiency and reliability of the monitoring system.

[0082] Among them, the multi-view and multi-directional fire change detection dataset FIGLib includes infrared images, satellite images, and meteorological data. This dataset can fully understand the development of fires and improve the accuracy of fire detection and prediction. It specifically includes the following steps:

[0083] The present invention selects a public dataset FIGLib, which reflects wildfire image sequences seen from fixed-view cameras located on remote mountaintops in Southern California. As of December 2021, the dataset consists of 315 fire sequences from 101 cameras at 30 sites between June 2016 and July 2021. Each sequence typically contains images 40 minutes before and 40 minutes after the start of the fire, as binary smoke / no smoke labels for each image, and is approximately 60 seconds apart, with a total of 81 images per fire sequence. In total, the dataset contains 24,800 high-resolution images with a size of 1536×2048 or 2048×3072 pixels, as well as associated meteorological data. Ignition detection and pre-ignition observations are achieved through camera cluster deployment, with four 90-degree or more viewing angles that can maintain consistency over many years, covering 360 degrees of the mountaintop.

[0084] The problem is that in wildfire detection, there are challenges with fixed perspectives and insufficient data, especially for such rare but extremely important events. Fixed perspectives and insufficient data may lead to insufficient model training, affecting its generalization and accuracy. Training data under such restrictive conditions may not cover a variety of scenarios and angles, limiting the model's comprehensive understanding of signs of fire. On the other hand, in nighttime environments, the light of fireflies can easily be misreported as dark fires, and frequent false alarms may reduce the reliability of fire detection models, making it difficult to identify real signs of fire, thereby affecting the efficiency of emergency response. This problem requires solving engineering problems, including how to distinguish between the light of fireflies and real signs of fire, in order to improve the accuracy and reliability of fire detection models.

[0085] The present invention proposes a wildfire prediction algorithm that integrates multi-source time series data suitable for edge computing, including:

[0086] Step 1: Use of multi-view and multi-directional dataset module;

[0087] This application uses a new multi-view and multi-directional fire change detection dataset FIGLib Dataset to verify the generalization performance of the network. This dataset FIGLib contains RGB images and infrared images of wildfires before and after the occurrence of different angles taken by cameras in different directions, as well as temperature, humidity, and wind meteorological data.

[0088] Step 1.1, select the FIGLib dataset, which includes RGB images and infrared images of wildfires at different angles taken by cameras in different directions, as well as relevant meteorological data such as temperature and humidity. The dataset can be represented as follows:

[0089]

[0090] in represents the original data set, express time Red picture from perspective, express time Green picture from perspective, express time Blue picture from perspective, express time The infrared image corresponding to the RGB image of the viewing angle, express Temperature data at the moment, express Wind data at the moment, express Humidity data at all times;

[0091] For example, taking the previous five moments and the five angles of view of a mountain and a drone as examples, the original data set There are 115 data in total, including 25 red pictures, 25 green pictures, and 25 blue pictures. , 25 infrared images , 5 temperature data , 5 wind speed and direction data , 5 humidity data .

[0092] Step 1.2 normalizes the temperature, humidity, and wind speed data in the data set. The following formula is used:

[0093]

[0094] The preprocessed data set is expressed as:

[0095] in for Normalization results of temperature data at each moment; for Normalization result of wind data at that time; for Normalization results of humidity data at all times; express Temperature data at the moment, express Wind data at the moment, express Humidity data at all times; is the minimum temperature, is the maximum temperature, is the minimum wind speed, is the maximum wind speed, is the minimum humidity value, is the maximum humidity;

[0096] Step 1.3, label or annotate the data, that is, assign the correct label or category to each sample. Use computer vision technology for automatic labeling. That is, use object detection, semantic segmentation and other technologies to detect the flame or smoke area in the image.

[0097] Step 1.4, divide the data set into training set, validation set and test set. The ratio used in this data set is 70% of the data for training, 15% of the data for validation, and 15% for testing.

[0098] Step 2: perform infrared data smoothing processing module.

[0099] In field fire detection, infrared sensors are often used to detect features such as heat and smoke generated by fires. However, in field environments, these sensors may be subject to various interferences, including weather changes, plant and animal activities, and changes in light. Especially at night, the light of fireflies can be easily misreported as a dark fire. Smoothing can help improve the stability and accuracy of infrared data, thereby improving the performance of fire detection systems.

[0100] Step 2.1: Use Gaussian filtering to weight the neighborhood around each data point so that the farther the data point is from the center point, the smaller the weight. Set a size of Gaussian kernel , the center point is , for infrared images Each pixel in is denoted as , the following formula is expressed:

[0101] ; ;

[0102] in, and is the index of the Gaussian kernel, and the range of the loop traversal is , is the pixel in the input image The gray value of is the pixel in the output image after convolution The value of is the standard deviation of the Gaussian kernel, is the base of natural logarithms;

[0103] Step 2.2, determine the window size. Determining the window size is one of the key steps in Gaussian smoothing. When determining the window size, it is necessary to consider the characteristics of the image, the noise level, and the application requirements. Considering that in order to solve the interference caused by the light of fireflies at night, a larger window needs to be selected. Here, a 7×7 window is selected to better smooth the image.

[0104] Step 3: Establish a new Smokey Net architecture module for outdoor fire smoke prediction, which specifically includes the following steps:

[0105] The Smokey Net architecture proposed in this paper is a new spatiotemporal grid image classification algorithm for wildfire smoke detection. After preparing RGB and infrared wildfire video sequence data and environmental data, the workflow of this architecture extracts image features through CNN and combines them with environmental data. The processed data is then input into LSTM for time series analysis to reveal the development trend of the fire, and finally achieve accurate detection of fire smoke. Figure 1 shown.

[0106] The Smokey Net architecture combines two different types of neural networks, two attention mechanisms, and an external register: the two different types of neural networks are CNN and LSTM, and the two attention mechanisms are Multimodal Sparse Transformer and Hard Attention. Figure 3 shown.

[0107] In this step, the Smokey Net architecture proposed in the present invention is the prediction core, and its purpose is to accurately predict wildfires as early as possible. For example, when a dark fire occurs, the smoke is not too dense and there is only a faint light, but the surrounding surface temperature rises, and the infrared image can detect it. Therefore, the Smokey Net architecture proposed in the present invention can predict wildfires in time and then minimize losses. The new SmokeyNet architecture suitable for edge computing proposed in the present invention is as follows:

[0108] Step 3.1 Choose a suitable CNN backbone network

[0109] The Smokey Net architecture proposed in the present invention first uses the CNN module to extract each frame of RGB three-color image and infrared image, and effectively extracts rich feature information from the RGB image and infrared image. These features include fire edges, fire textures, flame shapes, etc., which help to identify the visual characteristics of fire smoke and thus accurately predict wild fires. Considering that the data set constructed by the present invention is a small sample data set, a comprehensive comparison of multiple CNN networks such as ResNet34, MobileNet, EfficientNet, etc. was made. The present invention finally selected the VGG16 architecture containing 16 3x3 convolution kernels and convolutional layers and pooling layers with a stride of 1 as the CNN backbone network. The last layer of the second to last convolution block of VGG16, that is, the block5_conv3 layer, is selected as the feature extraction. The workflow of VGG16 is to input time Red picture from perspective , time Green picture from perspective , time Blue picture from perspective and infrared images The feature maps obtained after VGG16 are recorded as , the following formula is expressed:

[0110]

[0111] VGG16 network structure input layer, convolution layer, pooling layer, fully connected layer such as Figure 4 As shown. Among them:

[0112] The input layer is a three-channel image of 224*224*3. When processing infrared images, it needs to be converted into a single-channel image of 224*224*1.

[0113] The convolution layer contains 16 3x3 convolution kernels with a stride of 1 and a padding of 1. At the same time, each convolution layer is followed by a ReLU activation function to introduce nonlinear characteristics.

[0114] The pooling layer is placed after some convolutional layers to reduce the size of the feature map. The pooling kernel size is 2x2 and the stride is 2.

[0115] The fully connected layer mainly includes 3 fully connected layers. The number of neurons in the first two fully connected layers is 4096, and the last fully connected layer outputs the probabilities of 1000 categories.

[0116] The Softmax layer is mainly used to output classification probability.

[0117] Its workflow mainly uses convolution operations, which is a process of sliding a convolution kernel (also called a filter) on the input image, calculating the inner product between the convolution kernel and the local area of ​​the image, and obtaining a convolution feature map. Each element of the convolution kernel corresponds to a weight, and the correlation between the RGB three-color image and the infrared image can be extracted through different weight combinations. The pooling layer follows the convolution layer and is used to reduce the spatial dimension and number of parameters of the feature map, making the model more robust. Finally, by stacking multiple convolution layers and pooling layers, the abstract features of the data are gradually extracted, from low-level edges and textures to high-level shapes.

[0118] Step 3.2: Multi-source data fusion

[0119] The multi-source data fusion proposed in the present invention aims to comprehensively utilize RGB images and infrared image data, combined with environmental information such as temperature, humidity, wind and meteorological data, to improve the accuracy and efficiency of fire prediction, realize early detection, positioning and tracking of fires, so as to take timely measures to deal with fire crises and protect people’s lives and property. The multi-source data fusion workflow in the present invention is to fuse the feature images of each RGB three-color image and infrared image extracted by CNN with meteorological factors such as temperature, humidity and wind to generate a new feature image. First, assume that the three data sources of temperature, humidity and wind direction are in The weights of the moments are , then the formula of weighted fusion is expressed as:

[0120]

[0121] After the fusion is completed The characteristic of the moment is recorded as , the formula is:

[0122]

[0123] In summary, the data fusion involved in the present invention is mainly feature layer fusion. During the operation of the Smokey Net architecture, it is manifested as the feature picture of each frame of RGB three-color image and infrared image extracted by CNN, and then the bottom of the extracted feature picture is expanded by 30 rows respectively to save meteorological factors such as temperature and humidity in these 30 rows, and finally a new feature picture is generated.

[0124] Step 3.3 Select the attention mechanism module

[0125] The attention mechanism plays a vital role in the model. It allows the model to focus on relevant parts when processing sequence data, thereby improving the model performance. It is difficult to achieve the desired requirements by adopting only one attention mechanism. Therefore, the present invention selects two attention mechanisms, sparse attention mechanism (Sparse Attention), strong attention mechanism (Hard Attention) and multi-head attention mechanism (Multi-Head Attention). When the sparse attention mechanism is selected to process RGB and infrared pixel representation, the amount of calculation can be reduced, thereby reducing the computational complexity. When strong attention is selected to process meteorological data, ensure that the impact of key meteorological features on wildfire detection is emphasized. Selecting a multi-head attention mechanism to process RGB and infrared feature pixels and meteorological data improves the expression and generalization capabilities of the model.

[0126] First, the principle of the sparse attention mechanism is as follows: select the feature pixels of the RGB three-color time-series image and the infrared time-series image, and take the first pixel of the red R in the RGB three-color time-series image as the first pixel. For example, is the first Moment elements, remember for F 1 , so the Elements The time series is the input sequence represented by , is the length of the sequence. First, the similarity is calculated. For each position Calculate it with other positions The correlation between them is measured by inner product, Euclidean distance or cosine similarity. This process is a black box calculation inside the attention mechanism. According to the calculated similarity, the The most relevant positions form a sparse subset The method used is to select according to the similarity threshold, for each position , using the selected sparse subset Position in , the attention weight is calculated by the softmax function ,in The formula is as follows:

[0127] ;

[0128] in Indicates location and The similarity between is the temperature parameter used to control the smoothness of the distribution,

[0129] Finally, according to the obtained attention weight , get the location Output characteristics , the formula is:

[0130] ;

[0131] The principle of the hard attention mechanism is as follows: in the temperature data, select the first moments, of which is the first Moment elements, remember for , then its input sequence , is the length of the sequence, at each time step , Hard Attention will choose a position according to the greedy strategy , as the focus of attention at the current time step, and then according to the selected position , calculate the attention weight vector of the current time step , where only the position The weight of the position is 1, the weight of other positions is 0, and the attention weight vector is used Perform weighted summation on the value vector to obtain the output vector for the current time step , the formula is as follows:

[0132] ;

[0133] Finally, the output vector sequence is obtained based on the position selected at each time step .

[0134] The following is the formula of Multi-head Attention. Its overall structure is as follows Figure 5 As shown:

[0135] The principle of the Multi-Head Attention mechanism is as follows: select the feature pixels of the RGB three-color image and the infrared image and the first moments, of which is the first Moment elements, remember for , then its input sequence is:

[0136] ;

[0137] is the length of the sequence. First, we pass the input sequence Multiply by the query matrix , key matrix Sum Matrix , get the query ,key Sum , then split the query, key, and value into Head, get ,in Indicates For each head , calculate the attention score , the formula is as follows:

[0138] ;

[0139] in, is the dimension of the key;

[0140] Compute the attention output of each head , the formula is as follows:

[0141]

[0142] Finally, the attention outputs of all heads are concatenated and multiplied by the output matrix The final output is shown in the following formula:

[0143]

[0144] Step 3.4 Add external register module

[0145] An external register is added to the model to store additional feature images. The main working process is to regularly evaluate the performance of the model during training, such as Figure 6 As shown. If the model performance is improved, the current feature image is selectively stored in the external register to replace the old feature image, and one group is updated each time to ensure that there are always 10 groups of data in the external register. Finally, in the forward propagation process of the model, the current input data and the feature image in the external register are combined. It is hoped that this method can enhance the expressiveness of the model and improve its generalization ability.

[0146] Step 3.5 Add Long Short-Term Memory Network (LSTM Neural Network)

[0147] The architecture of the long short-term memory network (LSTM neural network) is as follows Figure 7As shown in the figure, by introducing memory cells and gating mechanisms to improve the ability to learn features from long input sequences, the processed fusion data and the 10 sets of data stored in the external register are processed together in the LSTM. Through the design of the gating unit, the LSTM can effectively capture long-term dependencies and avoid the problem of gradient disappearance. This is particularly important for fire detection that requires considering data with a long time span, so it can improve the efficiency and accuracy of the fire detection system. The calculation formula of the LSTM state at a certain moment is as follows:

[0148]

[0149] Where: is the input vector; For the forgotten door at the moment The activation vector of For the input gate The activation vector at the moment; For the output gate The activation vector at the moment; for The memory cell activation vector at the moment; for The hidden state activation vector at time instant; is the input kernel of each gate; is the loop kernel of each gate; for deviation; is the Sigmoid activation function; tanh is the hyperbolic tangent activation function.

[0150] Step 4: training the Smokey Net architecture proposed in the present invention

[0151] Set the training parameters:

[0152] In order to control the model learning process during model training to achieve the best performance and effect. The convergence speed and generalization ability of the model are optimized by setting relevant parameters. Stochastic Gradient Descent (SGD) is used to update the model weights, determine the most appropriate learning rate, and continuously adjust the loss function and optimizer settings to achieve the ideal model training effect. The updated learning rate As shown below

[0153]

[0154] In the above formula, the initial momentum power is set to 0.8 and the initial learning rate is set to is 0.001, and uses represents the maximum number of iterations, Indicates the current iteration number.

[0155] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

[0156] By implementing the present invention, users can solve two key engineering problems in wildfire prediction:

[0157] (1) Wildfires generally last for a long time. It takes dozens of hours from the generation of sparks to the formation of obvious smoke and open flames. Obviously, it is not suitable to use the equal-interval sampling method for their detection. How to analyze data under the premise of long-term monitoring is a major engineering problem.

[0158] (2) Problems of insufficient detection perspective and data samples. In wildfire detection, fixed detection perspective and insufficient data samples are important issues, especially for rare but extremely important events such as wildfires. This will lead to insufficient model training and affect the generalization ability and accuracy of the model.

[0159] (3) At night, the brightness of fireflies can easily be misreported as dark fires. Frequent false alarms may reduce the reliability of fire detection models, making it difficult to identify real signs of fire, thereby affecting the efficiency of emergency response. This is also a major engineering problem.

[0160] This can solve the problem of insufficient sample data when predicting wildfires and improve the accuracy of long-term monitoring, thereby helping to reduce casualties and property losses caused by fires and improve the efficiency and timeliness of fire response.

Claims

1. A fire prediction method for edge computing that integrates multi-source time series data, characterized in that: The steps include: Step 1: Collect multi-view RGB images, multi-view infrared images and meteorological data of the target area, and pre-process the multi-view RGB images and meteorological data; Step 2, performing Gaussian smoothing on the multi-view infrared images; Step 3: Use CNN as the backbone network to build a Smokey Net prediction model, input the preprocessed multi-view RGB images and meteorological data, and the multi-view infrared images after Gaussian smoothing into the Smokey Net prediction model to perform multi-source data fusion; Step 4: Select three attention mechanisms: sparse attention mechanism, strong attention mechanism, and multi-head attention mechanism; Step 4 is as follows: The principle of the sparse attention mechanism is as follows: select the feature pixels of the RGB three-color time series image and the infrared time series image, and take the i-th element of the red R in the RGB three-color time series image as an example, where is the i-th element of the input sequence at the t-th moment, is F1, so the n time series of the ith element is the input sequence represented by F i =[F1,F2,…,F n ], n is the length of the sequence. The first step is to calculate the similarity. For each position k∈n, the correlation between it and other positions l∈n is calculated, measured by inner product, Euclidean distance or cosine similarity. This process is a black box calculation inside the attention mechanism. According to the calculated similarity, some positions that are most relevant to each position k are selected to form a sparse subset S k The method used is to select according to the similarity threshold. For each position k, the selected sparse subset S k At position k in the graph, the attention weight a is calculated by the softmax function. kl , where a kl The formula is as follows: where sin(x k ,x l ) represents the similarity between positions k and l, τ is the temperature parameter used to control the smoothness of the distribution, and finally according to the obtained attention weight a kl , get the output feature Attention (F k ,F i ), whose formula is: The principle of the hard attention mechanism is as follows: in the meteorological data, the tth moment is selected, where is the i-th element of the input sequence at the t-th moment, is T1, then its input sequence T=[T1,T2,…,T n ], n is the length of the sequence; At each time step t, HardAttention selects a position i according to the greedy strategy t , as the focus of attention at the current time step, and then according to the selected position i t , calculate the attention weight vector a of the current time step t , where only position i t The weight of the position is 1, the weight of other positions is 0, and the attention weight vector a is used t Perform weighted summation on the value vector to obtain the output vector y for the current time step t , the formula is as follows: y t =a t T=T it Finally, the output vector sequence Y=[y1,y2,…,y t ]; The principle of the multi-head attention mechanism is as follows: select the feature pixels of the RGB three-color image and the infrared image and the t-th moment of the meteorological data, where is the i-th element of the input sequence at the t-th moment, is F1, then its input sequence is: F t =[F1,F2,…,F n ] n is the length of the sequence. First, we multiply the input sequence X by the query matrix W. Q , key matrix W K Sum matrix W V , we get the query Q = XW Q , key K = XW K The sum V = XW V Then, the query, key and value are split into h heads respectively to get Q i ,K i ,V i , where i represents the i-th head, for each head i, calculate the attention score A i , the formula is as follows: Among them, d k is the dimension of the key; Calculate the attention output O of each head i , the formula is as follows: About i =A i In i Finally, the attention outputs of all heads are concatenated and multiplied by the output matrix W O The final output is shown in the following formula: MultiHeadAttention(X)=Concat(O1,O2,…,O h )W O Step 5: Add external register module and long short-term memory network; Step 6: Train the SmokeyNet prediction model, update the model weights through stochastic gradient descent, and adjust the loss function and optimizer to obtain the trained SmokeyNet prediction model. Use the trained Smokey Net prediction model to predict fires in the target area.

2. According to claim 1, a fire prediction method for fusing multi-source time series data suitable for edge computing is characterized in that: Step 1 specifically includes the following steps: Step 1.1: Select data from different sources, including RGB images of wildfires at different angles taken by cameras in different directions, infrared images, and relevant meteorological data on temperature and humidity. The original data set is represented as follows: Where D represents the original data set, The red picture represents the perspective i at time t, represents the green picture from perspective i at time t, The blue picture represents the perspective i at time t, Indicates the infrared image corresponding to the RGB image of the i-view at time t, t t represents the temperature data at time t, w t represents the wind speed data at time t, h t Represents the humidity data at time t; Step 1.2: Preprocess the temperature, humidity, and wind data in the data set to make their range consistent with the image pixel range of 0-255 to reduce the weight bias problem in the model training process. The preprocessing method for temperature, wind, and humidity data is normalization, which is expressed by the following formula: The preprocessed data set is expressed as: Where T t is the normalized result of temperature data at time t; W t is the normalized result of wind data at time t; H t is the normalized result of humidity data at time t; t t represents the temperature data at time t, w t represents the wind speed data at time t, h t Represents the humidity data at time t; t min is the minimum temperature, t max is the maximum temperature, w min is the minimum wind force, w max is the maximum wind speed, h min is the minimum humidity, h max is the maximum humidity; Step 1.3: Label or annotate the data, that is, assign the correct label or category to each sample, and use computer vision technology to automatically label, that is, use object detection and semantic segmentation technology to detect the flame or smoke area in the image; Step 1.4: Divide the dataset into training set, validation set and test set.

3. According to the fire prediction method of the fusion of multi-source time series data suitable for edge computing in claim 1, it is characterized in that: Step 2 specifically includes the following steps: Step 2.1: Use Gaussian filtering to weight the neighborhood around each data point so that the farther the data point is from the center point, the smaller the weight. Suppose a Gaussian kernel kernel(m,n) with a size of (2k+1)×(2k+1) and a center point (i,j). For the infrared image Y t i Each pixel in is denoted as The following formula represents: Among them, m and n are the indices of the Gaussian kernel, and the loop traverses the range of (-k, k). is the pixel in the input image The gray value of is the pixel in the output image after convolution The value of , σ is the standard deviation of the Gaussian kernel, and e is the base of the natural logarithm; Step 2.2, determine the window size. Considering the characteristics of the image, the noise level, and the application requirements, a 7×7 window is selected to better smooth the image.

4. According to the fire prediction method of the fusion of multi-source time series data suitable for edge computing in claim 1, it is characterized in that: Step 3 specifically includes the following steps: Step 3.1, use CNN to extract each frame of RGB three-color image and infrared image respectively, select the last layer of the penultimate convolution block of VGG16, that is, block5_conv3 layer, for feature extraction. The workflow of VGG16 is to input the red image of the i-view at time t Green picture from perspective i at time t Blue picture from perspective i at time t and infrared image y t The feature maps obtained after VGG16 are recorded as The following formula represents: Step 3.2: Perform multi-source data fusion, fuse the feature images of each RGB three-color image and infrared image extracted by CNN with the meteorological factors of temperature, humidity and wind to generate a new feature image. First, assume that the weights of the three data sources of temperature, humidity and wind direction at time t are The formula of weighted fusion is expressed as: The feature at time t after the fusion is completed is recorded as F t i , the formula is:

5. According to the fire prediction method of the fusion of multi-source time series data suitable for edge computing in claim 1, it is characterized in that: Step 5 specifically includes the following steps: Step 5.1, add external registers, use external registers to store additional feature images and continuously update to store feature images, thereby enhancing its performance and generalization ability. The working process is to selectively store the current feature images in the external register during training, replace the old feature images, and update 1 group each time to ensure that there are always 10 groups of data in the external register; finally, in the forward propagation process of the model, combine the current input data and the feature images in the external register; Step 5.2: Introduce the LSTM framework, where the calculation formula for the LSTM state at time t is: f t =σ(W f F t i +U f h t-1 +b f ) c j t =σ(W j F t i +U j h t-1 +b j ) o t =σ(W o F t i +U o h t-1 +b o ) c t =f t ·c t-1 +j t ·tanh(W c x t +U c h t-1 )+b c h t =o t ·tanh(c t ) Where: F t i is the input vector; f t is the activation vector of the forget gate at time t; j t is the activation vector of the input gate at time t; o t is the activation vector of the output gate at time t; c t is the memory cell activation vector at time t; h t is the hidden state activation vector at time t; W f , W j , W o is the input kernel of each gate; U f , U j , U o is the loop kernel of each gate; b f , b j , b o , b c is the deviation; σ is the Sigmoid activation function; tanh is the hyperbolic tangent activation function.

6. The fire prediction method for fusing multi-source time series data suitable for edge computing according to claim 1 is characterized in that: In step 6, the model weights are updated by stochastic gradient descent as follows: determine the learning rate, continuously adjust the loss function and optimizer settings, and update the learning rate l R As shown below In the above formula, the initial momentum power is set to 0.8 and the initial learning rate is set to l r is 0.001, and uses i max Indicates the maximum number of iterations, i pr Indicates the current iteration number.

7. A fire prediction system integrating multi-source time series data suitable for edge computing, used to implement the method according to claim 1, characterized in that: It includes multi-view and multi-directional data module, infrared data smoothing processing module, new SmokeyNet architecture module and training parameter setting module; The multi-view and multi-orientation data module includes RGB images and infrared images of different angles before and after a wildfire, as well as temperature, humidity, and wind meteorological data taken by cameras in different orientations; this module is used to provide more comprehensive and diverse information, improve the robustness of the system, and make it better adaptable to different environmental conditions and fire characteristics; The infrared data smoothing processing module is used to remove the nighttime interfering light sources contained in the infrared image; The novel Smokey Net architecture module combines CNN, LSTM and multi-modal sparse transformer. The Smokey Net architecture can fully combine the input RGB images, infrared images, and temperature, humidity, wind and meteorological data to fully utilize the information of different types of data, accurately detect wildfire smoke and predict the development trend of wildfires; The training parameter setting module adjusts relevant parameters to improve the training efficiency, generalization ability and performance of the field fire detection model.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

Images