Fire situation prediction method and system based on multi-source data

Abstract

The invention provides a fire situation prediction method based on multi-source data, and relates to the technical field of fire prediction. Firstly, fire image data and fire time sequence image data are collected and preprocessed; constructing and training an improved GhostNetV2 feature extraction network, inputting the fire image data into the trained improved GhostNetV2 feature extraction network, and extracting a fire image data time sequence feature matrix; preprocessing the extracted time sequence characteristic matrix and the fire time sequence image data to obtain a time sequence sample; finally, a TSF fire prediction network is constructed and trained, and the TSF fire prediction network comprises a plurality of decision trees; and inputting the pre-time sequence sample into the trained TSF fire prediction network, and predicting a fire situation based on an output result of the decision tree. According to the fire situation prediction method provided by the invention, the accuracy and timeliness of the fire situation prediction method are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fire prediction, and more specifically relates to a fire situation prediction method and system based on multi-source data. Background Art

[0002] Disasters such as forest fires and urban fires severely disrupt ecosystem balance and cause immeasurable damage to human life and property. Traditional fire monitoring methods, such as manual patrols, require significant manpower and material resources and are inefficient. Furthermore, due to geographical and weather conditions, manual patrols cannot fully cover large areas, and are particularly ineffective in remote and complex terrain. While fixed cameras can partially compensate for the shortcomings of manual patrols, their fixed viewing angles create numerous blind spots, making it difficult to fully capture the occurrence and development of fires.

[0003] With the rapid development of drone technology, using drones equipped with image acquisition equipment for fire monitoring has become a research hotspot and development trend in the field of fire prevention and control. Drones offer high maneuverability and a wide monitoring range, enabling them to quickly reach fire scenes and acquire real-time image information. However, existing fire situation prediction technologies based on drone imagery still face numerous challenges. In complex natural and urban environments, the scale of flames and smoke in images varies greatly, from fine smoke produced by weak fire sources to raging flames and dense smoke from large-scale fires. This scale difference greatly complicates accurate feature extraction. Furthermore, complex backgrounds, such as dense forest vegetation and urban buildings, as well as variable lighting conditions, such as strong sunlight during the day and low light at night, can significantly interfere with the extraction of flame and smoke features, reducing their accuracy. Subsequent processing of real-time image data for fire prediction cannot meet the requirements for rapid response, resulting in insufficient timeliness and accuracy in fire situation prediction. Summary of the Invention

[0004] In order to solve the problem of insufficient timeliness and accuracy of existing fire situation prediction methods, the present invention proposes a fire situation prediction method and system based on multi-source data to improve the accuracy and timeliness of the fire situation prediction method.

[0005] In order to achieve the above technical effects, the technical solutions of the present invention are as follows: In a first aspect, the present invention provides a fire situation prediction method based on multi-source data, comprising the following steps: S1: Collect fire image data and fire time series image data and perform preprocessing; S2: Build and train an improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; S3: Preprocess the extracted time series feature matrix and fire time series image data to obtain time series samples; S4: Construct and train a TSF fire prediction network, which includes several decision trees; input time series samples into the trained TSF fire prediction network, and predict the fire situation based on the output results of the decision trees.

[0006] Furthermore, using drones to collect fire image data of different scene environments, different times, and different stages of fire development; marking the flame area and smoke area in the fire image data; At different stages of continuous fire development, drones are used to collect fire images to obtain fire image data, and fire time series data including temperature and smoke concentration environmental parameters are obtained. Based on the fire image data and fire time series data, fire time series image data is constructed; the fire time series image data includes: numerical characteristics of temperature and smoke concentration and fire image characteristics.

[0007] Furthermore, the improved GhostNetV2 feature extraction network includes: a first Ghost module for extracting fire image features, a first DFC module for generating a fire image attention map, a first convolution module for feature fusion, and a second Ghost module for extracting multi-scale features; the output ends of the first Ghost module and the first DFC module are both connected to the input end of the first convolution module, and the output end of the first convolution module is connected to the input end of the second Ghost module; Any Ghost module in the first Ghost module and the second Ghost module includes: a second convolutional layer and a first cheap operation layer connected in sequence; The first DFC module includes: a third convolutional layer, a first horizontal fully connected layer, and a first vertical fully connected layer connected in sequence.

[0008] Furthermore, in any one of the first Ghost module and the second Ghost module: Represents the input fire image data, and the intrinsic features of the fire image data are extracted through convolution operation. The expression is:

[0009] Where, represents the extracted intrinsic features, Indicates that the convolution kernel size is 1 1 convolution operation; Perform cheap operations on intrinsic features to extract deep features, and then concatenate the intrinsic features and deep features to obtain concatenated features. The expression is:

[0010] In the formula, Y represents the concatenated feature, Concat represents the concatenation operation, Represents deep features that can be extracted with cheap operations.

[0011] Furthermore, in the first DFC module: the input fire image data is transformed into Convert to feature map , the feature map Perform horizontal full connection and vertical full connection calculations to generate fire image attention maps .

[0012] Furthermore, when training the improved GhostNetV2 feature extraction network, a contrast loss function is designed. When the contrast loss function converges, the trained improved GhostNetV2 feature extraction network is obtained. The expression of the contrast loss function is:

[0013] Where, L represents the loss function, represents the category label, represents the feature distance, Represents the marginal threshold.

[0014] Based on the above technical means, the improved GhostNetV2 feature extraction network is adopted to extract the time series feature matrix, which can efficiently extract fire-related features in the fire situation prediction scenario and improve the accuracy of fire situation prediction.

[0015] Furthermore, it also includes: introducing a hybrid attention mechanism to map the features in the extracted fire image data time series feature matrix to different low-dimensional subspaces, each low-dimensional subspace uses an independent attention head, and uses the attention head to calculate the attention weights of different feature dimensions, and adjust the weights of different features in the fire image data time series feature matrix according to the attention weights.

[0016] Furthermore, the extracted time series feature matrix and the fire time series image data are spliced ​​and converted into time series samples; the time series samples are represented as The matrix, represents the number of time steps, Represents the dimension of the feature vector, which includes time series features and image features. 2 represents the numerical features of temperature and smoke concentration in the fire time series image data. The expression is:

[0017] Where, W Shows a time series sample, Indicates the The time step eigenvalues.

[0018] Furthermore, the process of constructing the TSF fire prediction network and outputting the predicted fire situation includes: Randomly sample time points and select the number of feature dimensions from the input time series samples, and construct K decision trees through recursive partitioning based on the randomly sampled time points and feature dimensions; The leaf nodes of the constructed decision tree output the mean value of the fire probability, and the output results of K decision trees are integrated to predict the fire situation; The expression for predicting fire situation is:

[0019] Where, represents the number of decision trees, Indicates the A decision tree for the input sample The prediction results, Represents the result of predicting the fire situation.

[0020] Based on the above technical means, a TSF fire prediction network is constructed to capture the characteristic information of fires in different time periods. By integrating the results of multiple decision trees, the accuracy and stability of fire situation prediction are improved.

[0021] On the other hand, the present invention also provides a fire situation prediction system based on multi-source data, comprising: Data acquisition module, used to collect fire image data and fire time series image data and perform preprocessing; The feature extraction module is used to build and train the improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; The data processing module is used to pre-process the extracted time series feature matrix and fire time series image data to obtain time series samples; The fire situation prediction module is used to build and train the TSF fire prediction network, which includes several decision trees; input time series samples into the trained TSF fire prediction network, and predict the fire situation based on the output results of the decision trees.

[0022] Compared with the prior art, the beneficial effects of this method are: The present invention provides a fire situation prediction method and system based on multi-source data. First, fire image data and fire time series image data are collected and preprocessed, and an improved GhostNetV2 feature extraction network is constructed and trained to extract the fire image time series feature matrix. Then, both the time series feature matrix and the fire time series image data are preprocessed to obtain time series samples. Finally, a TSF fire prediction network containing a decision tree is constructed and trained to complete the situation prediction. The present invention extracts fire image features through an improved GhostNetV2 feature extraction network, and effectively processes time series samples using a TSF network constructed using a decision tree, thereby improving the accuracy of fire feature extraction and enhancing the timeliness of fire situation prediction, providing accurate data support for fire prevention and control and emergency response. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart showing a fire situation prediction method based on multi-source data proposed in an embodiment of the present invention; Figure 2 A structural diagram showing the improved GhostNetV2 feature extraction network proposed in an embodiment of the present invention; Figure 3 The figure shows the structure of the fire situation prediction system based on multi-source data proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent; In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size; It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0025] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0026] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent; Example 1 In this embodiment, a fire situation prediction method based on multi-source data is proposed. Figure 1 The flowchart shown includes the following steps: S1: Collect fire image data and fire time series image data and perform preprocessing; S2: Build and train an improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; S3: Preprocess the extracted time series feature matrix and fire time series image data to obtain time series samples; S4: Construct and train a TSF fire prediction network, which includes several decision trees; input time series samples into the trained TSF fire prediction network, and predict the fire situation based on the output results of the decision trees.

[0027] Using drones to collect fire image data at different scene environments, at different times, and at different stages of fire development; marking flame areas and smoke areas in the fire image data; At different stages of continuous fire development, drones are used to collect fire images to obtain fire image data, and fire time series data including temperature and smoke concentration environmental parameters are obtained. Based on the fire image data and fire time series data, fire time series image data is constructed; the fire time series image data includes: numerical characteristics of temperature and smoke concentration and fire image characteristics.

[0028] like Figure 2 As shown in the structural diagram, the improved GhostNetV2 feature extraction network includes: a first Ghost module for extracting fire image features, a first DFC module for generating a fire image attention map, a first convolution module for feature fusion, and a second Ghost module for extracting multi-scale features; the output ends of the first Ghost module and the first DFC module are both connected to the input end of the first convolution module, and the output end of the first convolution module is connected to the input end of the second Ghost module; Any Ghost module in the first Ghost module and the second Ghost module includes: a second convolutional layer and a first cheap operation layer connected in sequence; The first DFC module includes: a third convolutional layer, a first horizontal fully connected layer, and a first vertical fully connected layer connected in sequence.

[0029] For example, according to different actual environments, the parameters of the Ghost module are determined, including the convolution kernel size and number of channels of point-by-point convolution and depth-wise convolution, so that it can efficiently extract basic features in fire situation prediction scenarios. In the DFC module, the number of neurons in the horizontal fully connected layer and the vertical fully connected layer, the weight initialization method, etc. are determined, and its ability to effectively capture the long-distance dependencies between flames and smoke while reducing computational complexity is verified through experiments. The bottleneck structure of the GhostNetV2 feature extraction network is optimized, and the specific connection method and parameter adjustment method of the DFC module in parallel with the first Ghost module are used to enhance the extended features generated by the first Ghost module and improve the model's ability to express fire characteristics.

[0030] In any one of the first Ghost module and the second Ghost module: Represents the input fire image data, and the intrinsic features of the fire image data are extracted through convolution operation. The expression is:

[0031] Where, represents the extracted intrinsic features, Indicates that the convolution kernel size is 1 1 convolution operation; Perform cheap operations on intrinsic features to extract deep features, and then concatenate the intrinsic features and deep features to obtain concatenated features. The expression is:

[0032] In the formula, Y represents the concatenated feature, Concat represents the concatenation operation, Represents deep features that can be extracted with cheap operations.

[0033] In the first DFC module: the input fire image data is transformed into Convert to feature map , the feature map Perform horizontal full connection and vertical full connection calculations to generate fire image attention maps .

[0034] Among them, the expression for horizontal full connection calculation is:

[0035] Where, Indicates that after the horizontal full connection calculation, the height is , with a width of The eigenvalue at position, represents the height of the feature map, represents the weight parameter of the horizontal fully connected layer, represents the multiplication operation, Represents the input feature map At a height of Width is The eigenvalue at the position; The expression for vertical full connection calculation is:

[0036] Where, Indicates that the height obtained after vertical full connection calculation is , with a width of The attention feature value at position, represents the width of the feature map, Represents the weight parameters of the vertical fully connected layer.

[0037] The Decoupled Fully Connected (DFC) attention mechanism in the DFC module addresses the limitations of lightweight convolutional neural networks in modeling long-range dependencies while maintaining efficient inference speed to meet the needs of mobile device applications. Due to the importance of long-range dependencies between flames and smoke in fire images for prediction, the fully connected layer of the DFC attention mechanism is decomposed into horizontal and vertical fully connected layers to reduce computational complexity. The DFC attention mechanism captures the long-range dependencies between pixels in different spatial locations of flames and smoke, improving prediction accuracy. It can also be implemented using convolution by sharing some transformation weights, avoiding time-consuming operations and improving inference speed.

[0038] In this embodiment, the Ghost module outputs the feature , the DFC attention module generates an attention map ,pass Convolution will Convert to DFC input , and finally output the fire image data time series feature matrix, the expression is:

[0039] In the formula, O represents the time series feature matrix of fire image data, sigmoid represents the Sigmoid function, and the attention map The value of is mapped to (0,1), indicating the attention weight; X represents the input feature map.

[0040] This method combines fire-related features extracted by the Ghost module with information about important regions of interest from the DFC attention mechanism. Because directly applying the sigmoid function to large-scale feature maps can cause latency, it is applied after feature downsampling. While attention map values ​​may not be entirely within the (0, 1) range, the impact on final performance is negligible. Furthermore, the inverted residual bottleneck structure of GhostNetV2 is optimized by running the DFC module in parallel with the first Ghost module. This enhances the expanded features generated by the first Ghost module and then feeds them into the second Ghost module to generate the final output features, improving the model's ability to represent fire characteristics.

[0041] For example, the improved GhostNetV2 feature extraction network is trained using stochastic gradient descent (SGD), with an initial learning rate of 0.001, which is decayed to 0.9 times every 5 epochs.

[0042] When training the improved GhostNetV2 feature extraction network, a contrast loss function is designed. When the contrast loss function converges, the trained improved GhostNetV2 feature extraction network is obtained. The expression of the contrast loss function is:

[0043] Where, L represents the loss function, represents the category label, represents the feature distance, Represents the marginal threshold. Contrastive Loss can bring similar features (flame / smoke) closer together and push background features further away.

[0044] The batch size is set to 32, the training cycle is 30 epochs, and based on NVIDIA RTX 3080 GPU acceleration, the improved GhostNetV2 feature extraction network is used to learn highly discriminative fire-related features.

[0045] In this embodiment, an improved GhostNetV2 feature extraction network is used to extract the time series feature matrix of the fire image data. The method also includes: introducing a hybrid attention mechanism to map the color, shape and other features in the extracted time series feature matrix of the fire image data to different low-dimensional subspaces. Each low-dimensional subspace uses an independent attention head, and the attention weights of different feature dimensions are calculated using the attention head. The weights of different features in the time series feature matrix of the fire image data are adjusted according to the attention weights.

[0046] In this example, attention weights are obtained by linearly transforming features, calculating their similarity scores with themselves and other features, and then performing Softmax normalization. These weights are optimized using extensive training data, enabling the GhostNetV2 feature extraction network to accurately highlight fire-related features across different feature dimensions, such as the orange-red color and irregular shapes of flames, while effectively suppressing background interference features like the blue sky and green trees, thereby improving the accuracy and reliability of fire situation prediction.

[0047] In this embodiment, the extracted time series feature matrix and the fire time series image data are spliced ​​and converted into time series samples; the time series samples are represented as The matrix, represents the number of time steps, Represents the dimension of the feature vector, which includes time series features and image features. 2 represents the numerical features of temperature and smoke concentration in the fire time series image data. The expression is:

[0048] Where, W Shows a time series sample, Indicates the The time step eigenvalues.

[0049] The numerical features of temperature and smoke concentration in the time series samples are standardized, and the expression is:

[0050] Where, represents the normalized numerical eigenvalue, represents a numerical eigenvalue, represents the mean, represents the standard deviation; Normalize the image features in the time series samples. The expression is:

[0051] Where, represents the normalized image feature value, represents the image feature value, Indicates the minimum value of the image feature value, Indicates the maximum value of the image feature value.

[0052] In this embodiment, the process of constructing the TSF fire prediction network and outputting the predicted fire situation includes: The algorithm randomly samples time points and selects the number of feature dimensions from the input time series. Based on these randomly sampled time points and feature dimensions, K decision trees are constructed through recursive partitioning. During the recursive partitioning process, the principle of minimizing the mean square error (MSE) is employed to select the optimal features and partitioning points at each node of the decision tree to minimize the mean square error after partitioning. Ultimately, the leaf nodes of the decision tree output the mean fire probability or the binary probability.

[0053] Suppose the sample set of the current node is , the sample set According to a certain characteristic and split point Split into two subnodes and , then the mean square error MSE after segmentation is expressed as:

[0054] In the formula, |S| represents the sample set The number of samples, Represents a sample The label value of and Represents child nodes respectively and The label mean, MSE represents the mean square error after segmentation.

[0055] The leaf nodes of the constructed decision tree output the mean value of the fire probability, and the output results of K decision trees are integrated to predict the fire situation. The expression for predicting the fire situation is:

[0056] Where, represents the number of decision trees, Indicates the A decision tree for the input sample The prediction results, Represents the result of predicting the fire situation. Integrating the results of multiple decision trees can reduce the error of a single decision tree and improve the overall prediction accuracy and stability of the TSF fire prediction network.

[0057] For example, the number of decision trees K is set between 100 and 200, that is, , which can improve the robustness and generalization ability of the model.

[0058] The number of feature dimensions of each decision tree Set to or between 2 and 5, i.e. or This can increase the diversity of the TSF fire prediction network and avoid all decision trees being trained based on the same features, so that the TSF fire prediction network can better adapt to different data characteristics and patterns.

[0059] The length of the time window for random sampling time points Set to 10-30 seconds, which corresponds to 10-30 frames in the time series sample, that is: Each decision tree randomly samples time steps of this length during training, which can capture the local temporal patterns in the time series samples.

[0060] When building a decision tree, the minimum number of samples for a leaf node is Set to 10, that is It can prevent the decision tree from overgrowing and overfitting. When the number of samples of a node is less than When , no splitting is performed, thereby ensuring that the learning of the TSF fire prediction network on the training data will not be too complicated, and improving the generalization ability of the TSF fire prediction network on unknown data.

[0061] Set the number of parallelism to -1 to use all CPU cores to train the decision tree in parallel. The TSF fire prediction network mainly relies on the CPU for calculation. Parallel training can significantly accelerate the training process and improve training efficiency. Assume the number of CPU cores is , then parallel training can be performed simultaneously A decision tree is built to shorten the training time.

[0062] In this embodiment, the training process of a single decision tree is: First, randomly sample from the input time series time points, of which ,For example . Assume that the time point set of the original time series sample is , the set of time points after sampling is ,in, ,and The random sampling of time points enables the decision tree to focus on different parts of the time series samples, thereby capturing the characteristic information of different time periods.

[0063] Random selection feature dimensions, which include some channels in the image features and may also include the dimensions of sensor data. Let the original feature dimension set be ,The selected feature dimension set is: ,in and By randomly selecting feature dimensions, we can increase the diversity of decision trees and avoid training all decision trees based on the same features.

[0064] During the training of the TSF fire prediction network, we set the batch size to 64-128 (adjusted based on memory availability), the number of training epochs to 50-100, and hardware acceleration based on CPU parallel training. We used 5-fold cross-validation, with a training set to validation set ratio of 8:2. We monitored the mean square error (MSE), precision, and recall on the validation set to prevent overfitting.

[0065] Example 2 This embodiment also provides a fire situation prediction system based on multi-source data, such as Figure 3 The structural diagram shown includes: Data acquisition module, used to collect fire image data and fire time series image data and perform preprocessing; The feature extraction module is used to build and train the improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; The data processing module is used to pre-process the extracted time series feature matrix and fire time series image data to obtain time series samples; The fire situation prediction module is used to build and train the TSF fire prediction network, which includes several decision trees; the preprocessed time series samples are input into the trained TSF fire prediction network, and the fire situation is predicted based on the output results of the decision trees.

[0066] Example 3 This embodiment describes in detail the process of collecting fire image data and fire time series image data and performing preprocessing in the fire situation prediction method based on multi-source data.

[0067] In this embodiment, drones are used to collect fire image data in different environments, at different times, and at different stages of fire development; flame and smoke areas in the fire image data are marked; drones are used to continuously collect fire image data at different stages of fire development and fire time series data including environmental parameters such as temperature and smoke concentration to construct fire time series image data; the fire time series image data includes: numerical features of temperature and smoke concentration and fire image features.

[0068] In this embodiment, a drone equipped with specialized equipment is used to conduct comprehensive and systematic image acquisition in a variety of environments, at different times, and during different fire development stages. A multimodal drone perception payload system is constructed, integrating a high-resolution visible light imaging module (resolution ≥3840×2160, dynamic range ≥12EV), a long-wave infrared thermal imaging module (resolution ≥640×512, temperature measurement accuracy ±2°C), and an environmental parameter sensing unit (including a thermistor with a temperature measurement accuracy of ±0.5°C and a combustible gas concentration detection sensor). This system utilizes a multi-sensor spatiotemporal registration mechanism to enable synchronous acquisition of heterogeneous data.

[0069] The present invention collects fire image data specifically by: establishing a multidimensional geographic spatial sampling network covering three typical fire-prone areas: forest ecological zones (including coniferous forests and broad-leaved forest biomes), urban built-up areas (including commercial / residential / industrial functional areas) and wilderness transition zones; constructing a meteorological-terrain coupled sampling matrix for forest areas, specifically including: meteorological dimension: establishing an illumination gradient of 10^5-10^1 The lux light intensity spectrum covers meteorological conditions such as sunny days (no cloud cover), cloudy days (cloud cover ≥80%), and rainfall intensity levels (0.1-50 mm / h); time series dimension: design the golden hour sampling window (30 minutes before sunrise to 2 hours after sunset) and moonlight illumination level (0.01-0.1 lux); vegetation parameters: formulate canopy cover density classification standards (sparse: ≤30%, medium: 30-60%, dense: >60%); terrain gradient: set a terrain sampling belt with a slope of 15°-45°, focusing on collecting fire data on characteristic terrain such as ridgelines (slope ≥25°) and valleys (relative height difference ≥50m). Data; construct a functional area fire feature library for urban areas, including: building fire classification model: establish a brick-concrete / frame heterogeneous structure building fire sample set, covering low-rise (≤6 floors), medium-rise (7-18 floors) and high-rise (≥19 floors) building fire scenarios; urban heat island effect sampling points: deploy differentiated monitoring nodes in commercial core areas (floor area ratio ≥3.0), residential clusters (floor area ratio 1.5-2.5) and industrial storage areas (hazardous chemical storage density classification); special fire scenario library: construct a solid waste combustion scenario (calorific value 16-25MJ / kg) and transportation hub fire (people density ≥2 people / ㎡) feature data set.

[0070] In this embodiment, the method for collecting fire image data includes: constructing a full-time illumination gradient sampling model, defining a dynamic illumination change monitoring window for the dawn-dusk transition period (1 hour before sunrise to 1 hour after sunrise), and establishing a continuous illumination attenuation model (10^3-10^5 lux) covering the typical daytime period with a solar altitude angle of 30°-90°; specially setting up a midday strong light interference test scene (sun altitude angle ≥75°, illumination ≥10^5 lux) and an oblique light environment simulation module (sun azimuth angle 120°-240°, altitude angle 15°-30°); constructing a dual-mode data collection mechanism for artificial lighting illumination grading (10-100 lux) and natural moonlight illumination level (≤0.1 lux) for night monitoring, and using a back-illuminated CMOS sensor (pixel size ≥2.4μm) and dual native ISO technology (sensitivity ≥ISO 25600) imaging device, combined with a time domain noise reduction algorithm (multi-frame synthesis ≥ 8 frames) to achieve low-light image capture with a signal-to-noise ratio of ≥ 65dB.

[0071] In this embodiment, the method for collecting fire time series data includes: establishing a quantitative classification model for fire stages, defining three-stage discrimination criteria: fire initiation stage (flame area ≤ 5 m², temperature gradient ≤ 50°C / m², smoke optical density ≤ 0.3), heat release acceleration stage (flame area 5-50 m², temperature gradient 50-200°C / m², smoke optical density 0.3-1.2), and full development stage (flame area ≥ 50 m², temperature gradient ≥ 200°C / m², smoke optical density ≥ 1.2); using a multimodal data spatiotemporal registration algorithm (synchronization error ≤ 10 ms, spatial registration accuracy ≤ 5 px) to implement a three-dimensional mapping model of temperature, smoke, and visual features, and generating a multimodal training sample set with time series labels, where the stage judgment labels include physical characteristic parameters such as the quantitative fire spread rate threshold (0.1-5 m / s) and the smoke optical density gradient (0.05-2.0 OD / mm²).

[0072] The process for annotating flame and smoke areas in fire image data is as follows: flame shape annotation is limited to within 5% of the actual shape pixel deviation; flame size annotation is limited to within 8% of the estimated actual area; and flame position annotation is accurate to the pixel, ensuring that the annotation position deviates no more than 3 pixels from the actual flame center. Smoke concentration annotation is performed according to strict color depth and transparency quantitative standards, and consistency between different annotators for the same smoke area concentration must exceed 90%; smoke spread annotation is limited to within 10% of the estimated actual spread radius. For infrared thermal image annotation, the fire source shape annotation error is limited to within 8% of the actual shape pixel deviation; the temperature distribution area demarcation boundary error is limited to within ±2°C pixel range; and the thermal radiation intensity trend annotation is required to maintain high consistency with the actual measured data, with an accuracy rate exceeding 95%.

[0073] Fire image data is also screened, ensuring visible light image resolution of no less than 4K (3840×2160 pixels) to clearly capture subtle features of flames and smoke, as well as details of the surrounding environment. Image brightness uniformity is maintained within ±10% to avoid difficulties in feature extraction due to uneven lighting. Color reproduction exceeds 95%, faithfully reflecting the color information of the fire scene. Infrared thermal image resolution is maintained at no less than 640×512 pixels, and temperature measurement accuracy is controlled within ±2°C to ensure that thermal images accurately reflect the temperature distribution of the fire source.

[0074] To ensure diversity in fire image data, the imagery of different environmental types, such as forests, cities, and wilderness, was balanced, with images of each environmental type comprising no less than 20% of the total dataset. Regarding the stages of fire development, the ratio of images collected during the initial, developing, and active phases was approximately 1:2:1, ensuring that characteristics of fires at different stages could be learned. In terms of time, the ratio of images collected during the day to those collected at night was 3:1, with images evenly distributed across different time periods. Images from each typical time period (e.g., early morning, midday, etc.) comprised no less than 10% of the total dataset.

[0075] During the collection of fire time-series image data, a multi-year periodic sampling plan (sampling period ≥ 12 months, 100% seasonal coverage, ≥ 95% climate type completeness) was developed. A fire event-triggered collection model was established (recording the entire process from the moment the thermal anomaly threshold was triggered until the fire was cleared). A fire-scale-adaptive collection frequency control strategy was developed: for large-scale fires (burned area ≥ 100 m²), a 5-10 minute sampling interval and a 50m spatial sampling density were used. For small-scale fires (burned area < 100 m²), a high-density sampling interval of 1-3 minutes and a 10m spatial grid were used.

[0076] At the same time, a normal environmental benchmark data collection mechanism is established (full parameter sampling ≥1 time per month, including typical time periods during the day and night), and a normal scene discrimination feature library is constructed through environmental baseline feature modeling (temperature fluctuation standard deviation ≤ 2°C, smoke concentration background value ≤ 5ppm).

[0077] Through the above methods, a dataset with a total sample size of ≥100,000 will be constructed, with a visible-light to infrared image pair ratio of 7:3. By geographic environment, forest ecological samples will account for 35% (coniferous forest to broadleaf forest ratio of 1:1.2), urban functional area samples will account for 40% (commercial / residential / industrial building ratio of 1.5:2:1), and wilderness transition zone samples will account for 25%. By fire evolution, samples will account for 20% of the initial fire phase (including temperature gradient ≤ 50°C / m), 50% of the developing phase (including smoke diffusion vector field data), 20% of the active phase (including thermal radiation flux ≥ 20kW / m²), and 10% of the normal environment baseline samples. Through an incremental expansion strategy (annual growth rate ≥200%), the goal is to increase the total sample size to 300,000, with cross-seasonal sample coverage reaching 100% and extreme climate event sample completeness reaching 95%. Through all-round and multi-angle collection and annotation methods, a rich and diverse dataset covering various practical situations is constructed to provide sufficient and high-quality data support for the subsequent training process.

[0078] Example 4 This embodiment proposes an actual fire situation prediction application, which uses the proposed fire situation prediction method and system based on multi-source data to predict the fire situation.

[0079] In this embodiment, the drone is equipped with dual thermal imaging and visual sensors to collect fire image data and time-series fire image data, providing comprehensive fire-related information. The drone also features an optimized image transmission module, ensuring stable and rapid transmission of real-time images of the monitored area to ground equipment.

[0080] As an example, the parameters of the improved GhostNetV2 feature extraction network are first set. The Ghost module uses 1×1 pointwise convolution, with the number of channels dynamically adjusted based on the input fire features. For example, when the number of input feature channels is C, the number of output channels of the pointwise convolution is set to C / 2 to reduce computational complexity while preserving key feature information. The Ghost module uses depthwise convolution with kernel sizes of 3×3 and 5×5 (dynamically switched based on feature map size), further reducing the number of parameters through grouped convolution. For example, for feature maps with a resolution ≥512×512, 3×3 depthwise convolution is preferred; for lower-resolution feature maps (e.g., 128×128), 5×5 depthwise convolution is used to expand the receptive field. The output features of the pointwise convolution are concatenated with the expanded features generated by the depthwise convolution along the channel dimension, with the total number of channels controlled to 1.5 times the number of input channels, balancing feature richness and computational efficiency.

[0081] In the DFC module, the weight matrix size of the horizontal fully connected layer is set to H×H (H is the feature map height) and implemented through depthwise convolution (kernel size H×1), reducing the computational complexity from O(H²W) to O(HW). The weight matrix size of the vertical fully connected layer is set to W×W (W is the feature map width) and implemented through depthwise convolution (kernel size 1×W), reducing the computational complexity from O(HW²) to O(HW). Xavier uniform weight initialization is used to ensure stable variance of activation values ​​across layers and prevent vanishing or exploding gradients.

[0082] Ablation experiments were conducted to compare the performance of the GhostNetV2 feature extraction network before and after the introduction of the DFC module, verifying its effectiveness in capturing the long-distance dependencies of flames (such as the direction of flame spread) and the path of smoke diffusion. This ensured that while the computational complexity was reduced by more than 40%, the prediction accuracy was improved by ≥5%.

[0083] For example, in the process of training the improved GhostNetV2 feature extraction network, the stochastic gradient descent (SGD) configuration is set, the momentum parameter is set to 0.9, and the decay coefficient is 0.0005 to accelerate convergence and suppress overfitting. The initial learning rate is set to 0.001, and the cosine annealing strategy is decayed every 5 epochs. The expression is:

[0084] in, (Total number of epochs).

[0085] Based on the NVIDIA RTX 3080 GPU, the batch size is set to 64 (dynamically adjusted to 32 when the video memory is insufficient), and mixed precision training technology (FP16) is used to reduce video memory usage and accelerate the training process.

[0086] Preferably, the difference features of adjacent frames (such as the flame area change rate and smoke concentration gradient) are introduced into the extracted fire image data temporal feature matrix to enhance the temporal dynamic feature expression capability. The expression is:

[0087] Where, f represents the original eigenvalue, Indicates the difference between adjacent frames.

[0088] For the missing points of sensor data (such as temperature and smoke concentration) in the fire time series image data, linear interpolation combined with historical mean method is used to fill in the missing points. The expression is:

[0089] Where, Represents the historical mean of the feature, balancing adjacent observations with the long-term trend. Represents the interpolated feature value.

[0090] The parameters of the TSF fire prediction network were set, with the number of decision trees set to 150. A grid search was performed to verify the optimal value within the range of 100-200. When the feature dimension D ≤ 10, a fixed number of 5 features was selected; when D > 10, D features were selected to improve the TSF fire prediction network's adaptability to high-dimensional image features. The window length was dynamically adjusted based on the fire stage: During the incipient fire stage (≤ 5 minutes), the window length was set to 10 seconds (10 frames) to capture rapidly changing microscale features. During the developing fire stage (5-30 minutes), the window length was set to 30 seconds (30 frames) to analyze mesoscale temporal patterns.

[0091] During the training of the TSF fire prediction network, the OpenMP parallel framework was used to group decision trees onto different CPU cores (e.g., an 8-core CPU could train 8 trees simultaneously), increasing training speed by approximately 6 times. Feature segmentation thresholds for each tree were pre-calculated and stored as a lookup table (LUT). During prediction, hashing was used to quickly locate segmentation points, keeping single-sample inference time under 20ms.

[0092] The batch size (64-128) is dynamically set based on CPU memory usage, and automatically halved when memory usage exceeds 80% to prevent program crashes. INT8 quantization technology is used to compress decision tree weights, reducing memory usage and accelerating computation, with accuracy loss limited to 2%.

[0093] For example, during training, the training and validation sets were divided to ensure that the sample ratios for each fire stage were consistent with the overall dataset (the ratio of initial stage, developing stage, and active stage was 1:2:1) to prevent data bias from affecting model generalization. A pruning strategy was implemented to prune branches with fewer than 10 leaf nodes. The Gini index gain of the decision tree was also calculated, and invalid branches with a gain less than 0.1 were removed to reduce complexity. Multiple evaluation metrics, including MSE, precision, recall, and the temporal consistency index (TCI), were used to calculate the variance of prediction results for five consecutive frames, with a threshold of 0.1, to ensure stable and reliable predictions. A Bayesian optimization algorithm was used to jointly optimize the number of decision trees (100-200), time window length (10-30 seconds), and number of random features (2-5). The objective function was to maximize the F1 score on the validation set. The optimal parameter combination was determined to be: 180 trees, 20-second window length, and 4 random features.

[0094] The ground equipment uses a high-performance server equipped with NVIDIA A100 GPUs to accelerate the inference process. After the drone transmits fire image data and fire time-series image data to the ground equipment, the ground equipment first inputs the fire image data into a trained improved GhostNetV2 feature extraction network to extract the fire image data's time-series feature matrix. This extraction time is kept within 0.1 seconds to meet real-time requirements. The extracted time-series feature matrix and fire time-series image data are then input into a trained TSF fire prediction network to predict the fire situation, also within 0.1 seconds.

[0095] The fire situation prediction threshold is set at 0.6. When the predicted result exceeds this threshold, the ground equipment immediately sends an alert to the relevant fire department via the 4G / 5G communication module. The alert includes key information such as the approximate location of the fire (obtained through the drone's positioning information) and the possible scale of the fire (estimated based on the characteristics of the flames and smoke). Simultaneously, images of the fire scene and related data are stored in the server database for subsequent analysis and processing.

[0096] An adaptive time window model based on the dynamic characteristics of fire evolution was constructed. During the fire initiation phase (fire growth rate ≤ 0.5 m² / s), a real-time data stream input mode was used, with simultaneous access to visible light imagery (resolution ≥ 4K - 60 fps), infrared thermal imagery (temperature resolution ± 2°C), and multi-source sensor data (temperature sampling rate ≥ 1 Hz, smoke concentration detection delay ≤ 3 s). During the fire acceleration phase (fire growth rate 0.5-5 m² / s), a 5-10 minute sliding time window was activated to construct a multimodal data fusion matrix (including visible light-infrared image sequences with a spatiotemporal registration error of ≤ 5 px and temperature-smoke concentration spatiotemporal evolution curves). A multi-dimensional prediction system for fire development in the next 1-2 hours was established, with outputs including: fire line spread vector field (direction prediction accuracy ≤15°, rate prediction error ≤0.3m / s); heat release rate growth model (power prediction error ≤15%); smoke diffusion impact domain (PM2.5 concentration distribution prediction resolution ≤50m); and final burned area probability cloud map (combustion boundary prediction zone with confidence ≥90%).

[0097] A time-coupled model between prediction results and emergency resource scheduling was established, with a focus on outputting the following during the golden disposal window (30-120 minutes after a disaster): fire rescue path optimization plan (path avoidance algorithm based on fire front prediction); evacuation channel dynamic programming model (travel time estimation combined with smoke diffusion prediction); and thermal radiation protection threshold for key protection targets (time margin prediction corresponding to the 10kW / m² warning line).

[0098] This solution achieves an organic connection between early warning (second-level response) and medium-term prediction (minute-level evolution) through an adaptive spatiotemporal data fusion strategy according to the fire stage. Its prediction time scale accurately matches the standard fire emergency response process (National Fire Emergency Rescue Specification GB / T 38315-2019), effectively solving the prediction lag problem caused by the fixed time window in traditional methods.

[0099] By providing prediction results 1-2 hours in advance, the fire department has enough time to formulate a scientific and reasonable fire-fighting operation plan, including determining the deployment plan of fire-fighting forces, planning rescue routes, preparing appropriate fire-fighting equipment and supplies, etc., thereby minimizing the loss of life and property and ecological damage caused by the fire.

[0100] Furthermore, a feedback mechanism is established. After receiving an alarm and handling a fire, the fire department will provide the system with actual fire conditions. Based on this feedback, the system fine-tunes the model to further improve prediction accuracy and reliability, continuously optimizing the fire situation prediction method and system performance.

[0101] For example, suppose a fire breaks out in a forest in a mountainous area. The area has dense vegetation, mostly coniferous forests, and complex terrain with a certain slope. The fire situation prediction process is as follows: Step 1: A sudden forest fire breaks out in a mountainous area. The drone responds quickly. Equipped with dual thermal imaging and visual sensors, it effectively captures fire-related information in complex environments. During flight, the drone adjusts its imaging parameters based on the fire's stage and environmental conditions. In the early stages of a fire, the drone operates in high-resolution, high-frame-rate mode with wide dynamic range (WDR) enabled to accurately capture subtle features of small fire sources and thin smoke. As the fire grows, it switches to wide-angle mode to capture a comprehensive view of the fire's spread. Simultaneously, the drone's onboard temperature and smoke concentration sensors collect real-time environmental data to assist in determining the fire's stage of development.

[0102] Step 2: The drone uses an optimized image transmission module to quickly and stably transmit real-time data to ground-based equipment. The ground-based equipment uses a high-performance server equipped with an NVIDIA A100 GPU. Upon receiving the fire image data, it immediately feeds the trained improved GhostNetV2 feature extraction network for feature extraction, with extraction time kept within 0.1 seconds, meeting real-time requirements. A hybrid attention mechanism is used to enhance features related to flames and smoke, mitigate background interference, and extract a temporal feature matrix from the fire image data.

[0103] Step 3: The extracted time series feature matrix and fire time series image data are fed into the trained TSF fire prediction network to predict the fire situation, again within 0.1 seconds. When the predicted fire situation exceeds a set threshold of 0.6, ground equipment quickly transmits an alert to the relevant fire department via 4G / 5G communication modules. This includes key information such as the approximate location of the fire (obtained from drone positioning information) and the potential size of the fire (estimated based on flame and smoke characteristics), providing crucial information for timely fire rescue efforts.

[0104] Step 4: Upon receiving the alarm, the fire department quickly dispatched to the scene to extinguish the fire. After the fire was extinguished, the actual fire situation was fed back to the system. Based on this feedback, the system fine-tuned the improved GhostNetV2 feature extraction network and the TSF fire prediction network. This enabled subsequent predictions to more accurately reflect similar fire scenarios, further improving prediction accuracy and reliability. This continuously optimized the fire situation prediction method and system performance to better prepare for future fires.

[0105] The embodiments are provided merely to illustrate the present invention and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications may be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the claims.

Claims

1. A fire situation prediction method based on multi-source data, characterized in that: The following steps are involved: Collect fire image data and fire time series image data and perform pre-processing; Build and train an improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; The extracted time series feature matrix and fire time series image data are preprocessed to obtain time series samples; A TSF fire prediction network is constructed and trained, wherein the TSF fire prediction network includes several decision trees; time series samples are input into the trained TSF fire prediction network, and the fire situation is predicted based on the output results of the decision trees.

2. A fire situation prediction method based on multi-source data according to claim 1, characterized in that: Using drones to collect fire image data at different scene environments, at different times, and at different stages of fire development; marking flame areas and smoke areas in the fire image data; At different stages of continuous fire development, drones are used to collect fire images to obtain fire image data, and fire time series data including temperature and smoke concentration environmental parameters are obtained. Based on the fire image data and fire time series data, fire time series image data is constructed; the fire time series image data includes: numerical characteristics of temperature and smoke concentration and fire image characteristics.

3. The fire situation prediction method based on multi-source data according to claim 1, characterized in that: The improved GhostNetV2 feature extraction network includes: a first Ghost module for extracting fire image features, a first DFC module for generating a fire image attention map, a first convolution module for feature fusion, and a second Ghost module for extracting multi-scale features; the output ends of the first Ghost module and the first DFC module are both connected to the input end of the first convolution module, and the output end of the first convolution module is connected to the input end of the second Ghost module; Any Ghost module in the first Ghost module and the second Ghost module includes: a second convolutional layer and a first cheap operation layer connected in sequence; The first DFC module includes: a third convolutional layer, a first horizontal fully connected layer, and a first vertical fully connected layer connected in sequence.

4. A fire situation prediction method based on multi-source data according to claim 3, characterized in that: In any one of the first Ghost module and the second Ghost module: Represents the input fire image data, and the intrinsic features of the fire image data are extracted through convolution operation. The expression is: Where, represents the extracted intrinsic features, Indicates that the convolution kernel size is 1 1 convolution operation; Perform cheap operations on intrinsic features to extract deep features, and then concatenate the intrinsic features and deep features to obtain concatenated features; The expression is: In the formula, Y represents the concatenated feature, Concat represents the concatenation operation, Represents deep features that can be extracted with cheap operations.

5. The fire situation prediction method based on multi-source data according to claim 3 is characterized in that: In the first DFC module: the input fire image data is transformed into Convert to feature map , the feature map Perform horizontal full connection and vertical full connection calculations to generate fire image attention maps .

6. The fire situation prediction method based on multi-source data according to claim 1, characterized in that: When training the improved GhostNetV2 feature extraction network, a contrast loss function is designed. When the contrast loss function converges, the trained improved GhostNetV2 feature extraction network is obtained. The expression of the contrast loss function is: Where, L represents the loss function, represents the category label, represents the feature distance, Represents the marginal threshold.

7. The fire situation prediction method based on multi-source data according to claim 1, characterized in that: Also includes: A hybrid attention mechanism is introduced to map the features in the extracted temporal feature matrix of fire image data into different low-dimensional subspaces. Each low-dimensional subspace adopts an independent attention head, and the attention weights of different feature dimensions are calculated by the attention head. The weights of different features in the temporal feature matrix of fire image data are adjusted according to the attention weights.

8. The fire situation prediction method based on multi-source data according to claim 2 is characterized in that: The extracted time series feature matrix and fire time series image data are spliced ​​and converted into time series samples; the time series samples are represented as The matrix, represents the number of time steps, Represents the dimension of the feature vector, which includes time series features and image features. 2 represents the numerical features of temperature and smoke concentration in the fire time series image data; The expression is: Where, W Shows a time series sample, Indicates the The time step eigenvalues.

9. The fire situation prediction method based on multi-source data according to claim 1, characterized in that: The process of building a TSF fire prediction network and outputting the predicted fire situation includes: Randomly sample time points and select the number of feature dimensions from the input time series samples, and construct K decision trees through recursive partitioning based on the randomly sampled time points and feature dimensions; The leaf nodes of the constructed decision tree output the mean value of the fire probability, and the output results of K decision trees are integrated to predict the fire situation; The expression for predicting fire situation is: Where, represents the number of decision trees, Indicates the A decision tree for the input sample The prediction results, Represents the result of predicting fire situation.

10. A fire situation prediction system based on multi-source data, used to implement the fire prediction method based on multi-source data according to any one of claims 1 to 9, characterized in that: include: Data acquisition module, used to collect fire image data and fire time series image data and perform preprocessing; The feature extraction module is used to build and train the improved GhostNetV2 feature extraction network, input the fire image data into the trained improved GhostNetV2 feature extraction network, and extract the time series feature matrix of the fire image data; The data processing module is used to pre-process the extracted time series feature matrix and fire time series image data to obtain time series samples; The fire situation prediction module is used to build and train the TSF fire prediction network, which includes several decision trees; input time series samples into the trained TSF fire prediction network, and predict the fire situation based on the output results of the decision trees.

Images