Mountain fire early warning method and system based on multi-view vision spatial-temporal characteristics
By employing a multi-view visual spatiotemporal feature method and utilizing stereo correction of primary and secondary cameras and multimodal feature fusion, the false alarm and accuracy problems of monocular cameras in wildfire monitoring were solved, enabling earlier and more accurate fire detection and three-dimensional position output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-10
- Publication Date
- 2026-03-24
AI Technical Summary
Existing methods for monitoring wildfires rely on monocular cameras, which are easily affected by environmental factors such as clouds, fog, and dust, leading to false alarms and an inability to accurately determine the scale and threat level of the fire. Furthermore, they are difficult to adapt to complex and ever-changing natural environments.
A multi-view vision spatiotemporal feature method is adopted, which simultaneously acquires images through main and secondary cameras, performs stereo correction and multimodal feature fusion, combines spatiotemporal verification and stereo matching, calculates a comprehensive confidence score, and generates a fire warning.
It reduced the false alarm rate of wildfire warnings, improved the accuracy and timeliness of warnings, and achieved precise spatial positioning and three-dimensional situational awareness of fires.
Smart Images
Figure CN121725564A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video surveillance technology, and in particular to a wildfire early warning method and system based on multi-view visual spatiotemporal characteristics. Background Technology
[0002] Currently, wildfire monitoring mainly relies on image analysis using monocular cameras to identify the color of smoke or flames (such as RGB, YC). b C r Early warning can be provided based on spatial and textural features (such as LBP).
[0003] However, this method has significant limitations. Specifically, environmental factors such as clouds, fog, and dust are similar in shape and color to smoke, which can easily lead to false alarms.
[0004] Furthermore, monocular images cannot accurately determine the true distance and three-dimensional size of a target. Smoke in the distance may be misjudged as a nearby fire source, and vice versa, thus making it impossible to accurately assess the true scale and threat level of a fire.
[0005] Meanwhile, traditional methods are mostly based on fixed thresholds or weights of single-frame images for judgment, which is difficult to adapt to complex and ever-changing natural environments (such as changes in lighting and dynamic backgrounds). These defects severely impact the accuracy and timeliness of wildfire warnings. Summary of the Invention
[0006] The technical problem this invention aims to solve is to provide a wildfire early warning method and system based on multi-view visual spatiotemporal features. This can reduce the false alarm rate of wildfire early warnings and improve their timeliness and accuracy.
[0007] To address the aforementioned technical problems, as one aspect of the present invention, a wildfire early warning method based on multi-view visual spatiotemporal features is provided, comprising the following steps: The system uses a main camera and a secondary camera deployed at the same horizontal level to simultaneously acquire images of the monitored scene; and performs stereoscopic correction processing on the images acquired by the main camera and the secondary camera. Multimodal visual features are extracted from the main camera image and adaptively fused to generate a comprehensive feature map; the comprehensive feature map is processed to obtain preliminary candidate fire areas; the candidate fire areas are subjected to two-dimensional spatiotemporal verification, which includes analyzing the persistence, growth trend and flicker frequency of the area; Based on the correspondence between the candidate fire areas in the corrected images of the main and secondary cameras, stereo matching and three-dimensional spatial positioning are performed, and the stereo verification confidence factor is calculated. Based on the multimodal visual feature fusion results, spatiotemporal verification results, and stereo verification confidence factor, a comprehensive confidence score is calculated for each candidate fire area; candidate fire areas with a comprehensive confidence score greater than a set score threshold are designated as target suspected fire areas, and an alarm is triggered.
[0008] Preferably, the extraction of multimodal visual features and adaptive fusion specifically includes: Extract the luminance component Y, red chromaticity component Cr, and blue chromaticity component Cb of each pixel from the main camera image; The Flame Saturation Index (FSI) of each pixel is calculated based on the luminance component Y, the red chromaticity component Cr, and the blue chromaticity component Cb, forming an FSI feature map. The FSI value of each pixel is calculated using the following formula:
[0009] in, This represents the average brightness value within a local neighborhood centered on that pixel. Extract the Cr feature map corresponding to the main camera image; Calculate the dynamic texture features of the main camera image and generate a texture change feature map; The FSI feature map, Cr feature map, and texture change feature map are input into a pre-trained weight adaptive model to obtain their respective dynamic weight maps. The three feature maps are weighted and fused according to the dynamic weight map to generate the comprehensive feature map.
[0010] Preferably, the step of calculating the dynamic texture features of the main camera image and generating a texture change feature map specifically includes: For each frame of the main camera image, the texture entropy of the local window where each pixel is located is calculated by a sliding window to generate a static texture feature map, wherein the texture entropy is calculated based on the local binary mode LBP histogram. Subtract the texture entropy value of the corresponding pixel in the static texture feature map of the current frame from the static texture feature map of the previous frame, and take the absolute value to obtain the texture change rate of each pixel, thereby generating the texture change feature map. The calculation process of the LBP value includes: Select a pixel in the image as the center; Take 8 pixels from its surrounding 3x3 neighborhood; The gray value of the center pixel is used as the threshold and compared with the gray values of the surrounding 8 pixels to generate an 8-bit binary number, which is then converted into a decimal number as the LBP value.
[0011] Preferably, in the spatiotemporal verification, the flicker frequency analysis specifically includes: The average brightness value of each candidate fire area over the past L consecutive frames is obtained to form a brightness time sequence signal; The power spectrum is obtained by performing a fast Fourier transform on the brightness time-series signal. The maximum power value of the power spectrum within a preset flame characteristic frequency range is calculated as the scintillation confidence level of that region, wherein the flame characteristic frequency range is from 1 Hz to 10 Hz.
[0012] Preferably, the stereo confidence factor is jointly determined by the confidence of stereo matching and the degree to which the three-dimensional dimensions of the candidate region conform to the flame model, and its value range is [0,1]. The formula for calculating the overall confidence score is as follows:
[0013] in, Let be the value of the i-th pixel in the candidate fire area on the comprehensive feature map, N be the total number of pixels in the area, G be the growth factor quantified according to the growth trend check results, Flicker_Confidence be the flicker confidence, and Stereo_Confidence be the stereo verification confidence factor.
[0014] Preferably, the weights are dynamically generated using a scene weight prediction model: Using the image brightness channel as input, a lightweight attention network trained under self-supervised guidance outputs the weight maps corresponding to the FSI feature map, Cr feature map, and texture change feature map. The goal of network training is to maximize the response of the fused integrated feature map in the real fire area and minimize the response in the background area. The loss function is either binary cross-entropy or mean squared error.
[0015] As another aspect of the present invention, a wildfire early warning system based on multi-view visual spatiotemporal features is also provided, comprising: The multi-view image acquisition module includes a main camera and a secondary camera with fixed spatial positions, used to synchronously acquire images of the monitored scene; The image processing and computing module is communicatively connected to the multi-view image acquisition module and is used to receive and process images; The image processing and calculation module is configured to perform: The images from the main and secondary cameras are subjected to stereo correction and matching to calculate depth information; Perform the multimodal feature extraction, fusion, candidate region extraction, and spatiotemporal verification steps in the aforementioned method based on the main camera image; The candidate fire areas are subjected to 3D matching and 3D localization, and the 3D verification confidence factor is calculated. The comprehensive confidence score is calculated based on the multimodal feature fusion results, spatiotemporal verification results, and stereo verification results, and an early warning signal is generated.
[0016] Preferably, the image processing calculation module further includes: The feature extraction and fusion unit is used to calculate the flame saturation index (FSI) feature map, red chromaticity (Cr) feature map, and dynamic texture change feature map of the input image; call the pre-trained weighted adaptive model to generate dynamic weights for the three feature maps; and perform weighted summation on the three feature maps according to the dynamic weights to output a comprehensive feature map. The spatiotemporal verification unit is used to maintain a historical state buffer for each candidate fire area to check the persistence and area growth trend of the area; a brightness time sequence buffer is established, and the flicker frequency confidence of the area is obtained by performing frequency domain analysis on the sequence; The stereo verification and positioning unit is used to perform stereo matching and three-dimensional spatial coordinate calculation for each candidate fire area by combining the secondary camera image, and to generate a stereo verification confidence factor based on the matching quality and three-dimensional size conformity.
[0017] The image processing computation module includes: the weight adaptive model is a scene-aware attention network trained based on self-supervised learning; the training objective of the network is to maximize the distinguishability of the comprehensive feature map obtained by fusing its output dynamic weights with the pre-computed FSI feature map, Cr feature map, and texture change feature map between the real fire area and the background area.
[0018] The image processing computing module includes: the image processing computing module is deployed in one of the following locations: a local processor integrated in the main camera or the secondary camera, an independent local processing center communicating with the camera, or a remote server in the cloud.
[0019] Implementing this embodiment has the following beneficial effects: This invention provides a wildfire early warning method and system based on multi-view visual spatiotemporal features. This invention employs stereo matching and 3D positioning using primary and secondary cameras. Unlike traditional monocular systems, it calculates the true distance and 3D size of the fire point through dual-camera parallax, effectively overcoming interference from two-dimensional similar objects such as clouds and fog, achieving precise spatial positioning of the fire, and providing crucial information for rescue and dispatch.
[0020] This invention employs an adaptive multi-feature fusion mechanism. It innovatively integrates flame saturation index, chromaticity features, and dynamic texture entropy, and utilizes an attention network to dynamically adjust the weights of each feature. This enables the system to adapt to changes in illumination and complex backgrounds, significantly improving detection robustness under different environments and reducing false alarms.
[0021] This invention employs a spatiotemporal dual verification mechanism. By introducing regional growth trend analysis and flame flashing frequency analysis, the persistence and dynamic characteristics of the fire are verified from a temporal perspective. This effectively filters out instantaneous interference such as flashes and reflections, improving the sensitivity of identifying real early-stage fires.
[0022] This invention employs a combined 3D and temporal confidence assessment. By integrating 3D verification confidence, growth factor, and scintillation confidence, a final alarm decision is generated, forming a multi-layered filtering approach that combines spatial, temporal, and feature-based methods. This approach ensures a high detection rate while achieving an extremely low false alarm rate.
[0023] These technologies work together to enable the system to detect fires earlier and more accurately, and to output the three-dimensional location and spread trend of the fire, truly achieving a leap from "two-dimensional alarm" to "three-dimensional situational awareness"; it can reduce false alarms, improve the accuracy of distance and size judgment, and enhance the timeliness and accuracy of wildfire early warning. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.
[0025] Figure 1 This is a schematic diagram of the main process of an embodiment of a wildfire early warning method based on multi-view visual spatiotemporal features provided by the present invention; Figure 2 This is a schematic diagram of the camera installation involved in the method provided by the present invention; Figure 3 This is a schematic diagram of the structure of an embodiment of a wildfire early warning system based on multi-view visual spatiotemporal features provided by the present invention; Figure 4 for Figure 3 A schematic diagram of the image processing and computing module. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.
[0027] like Figure 1 The diagram shown illustrates the main flow of an embodiment of a wildfire early warning method based on multi-view visual spatiotemporal features provided by the present invention; combined with... Figure 2 As shown, in this embodiment, the method includes at least the following steps: Step S1: Simultaneously acquire images of the monitored scene using a main camera and a secondary camera deployed at the same horizontal level, and perform stereo correction processing on the images acquired by the main camera and the secondary camera. Step S2: Extract multimodal visual features from the main camera image and perform adaptive fusion to generate a comprehensive feature map; process the comprehensive feature map to obtain preliminary candidate fire areas; perform two-dimensional spatiotemporal verification on the candidate fire areas, the spatiotemporal verification including analyzing the persistence, growth trend and flicker frequency of the area; Step S3: Based on the correspondence between the candidate fire areas in the corrected images of the main and secondary cameras, perform stereo matching and three-dimensional spatial positioning, and calculate the stereo verification confidence factor. Step S4: Calculate the comprehensive confidence score of each candidate fire area based on the multimodal visual feature fusion result, spatiotemporal verification result, and stereo verification confidence factor; designate candidate fire areas with comprehensive confidence scores greater than a set score threshold as target suspected fire areas and trigger an alarm.
[0028] The following provides a detailed description of each step involved in this method.
[0029] The system architecture involved in the method of this invention includes a main camera and a secondary camera, as well as an image processing and computing module. The main camera and the secondary camera are located at the same horizontal level and their functions are distributed accordingly, such as... Figure 2 As shown. The image processing computing module can be local, such as deployed in the main camera or the secondary camera, or deployed in a local processor that can communicate with the main camera or the secondary camera, or it can be in the cloud, for example, deployed in a remote server.
[0030] In step S1, images of the monitored scene are simultaneously acquired by a main camera and a secondary camera deployed at the same horizontal level. Stereo correction processing is then performed on the images acquired by the main and secondary cameras to eliminate distortion and align the rows of the two images, preparing for subsequent stereo matching. This step can be implemented using any relevant stereo correction technique.
[0031] In step S2, multimodal visual features are extracted from the main camera image and adaptively fused to generate a comprehensive feature map; the comprehensive feature map is processed to obtain preliminary candidate fire areas; and the candidate fire areas are subjected to two-dimensional spatiotemporal verification.
[0032] Specifically, in a particular example, the image captured by the main camera can be used as the primary analysis object, while the synchronized image from the secondary camera is retained for subsequent stereo matching. In other embodiments, the fused images from the main and secondary cameras can also be preprocessed; in the following example, the image captured by the main camera is used as the primary analysis object. Then, using a pre-trained fire detection model, suspected fire areas matching the color and texture characteristics of smoke or flames are initially detected, specifically including the following steps: Step S21: Construct a multimodal dynamic perception feature map.
[0033] The Y, Cb, and Cr components are extracted from the main camera image. Specifically, the Y component (luminance: representing the lightness or darkness of a pixel, without color information), Cr component (red chromaticity: representing the degree to which the pixel color deviates from red), and Cb component (blue chromaticity: representing the degree to which the pixel color deviates from blue) are defined in the color space. Since flame colors typically have high saturation, a Flame Saturation Index (FSI) related to the Y, Cr, and Cb components is defined. The FSI index for each pixel is calculated using the following formula:
[0034] Formula principle: Since the core of a flame is usually one of the brightest parts in an image, subtracting Y and Cb can effectively highlight pixels that are both bright and reddish. A red object in the shadow may have a high Cr value, but a low Y value. Therefore, the contribution of the Y-Cb term will be reduced, thereby lowering the FSI value and achieving adaptive suppression of illumination.
[0035] in, This is the average brightness of a local area of the image. A local area refers to a small neighborhood centered on the current pixel, such as a square window of size N×N pixels centered on the current pixel, where N is a preset odd number, such as 3, 5, or 7. In other transformation modes, the window size can be dynamically and adaptively adjusted, for example: First, calculate the pixel brightness variance within an initial window (e.g., 3×3). If the brightness variance is greater than a set threshold, it indicates that the texture of the area is complex, and the window is expanded to 7×7; otherwise, the initial window size is maintained.
[0036] Example: In flat areas, such as blue skies, use a small window for calculation; in complex areas, such as foliage, use a large window for calculation to obtain a more reliable background brightness estimate.
[0037] In addition, due to This represents the average brightness of the environment surrounding the current pixel. The formula above is essentially to determine how much redder a pixel is than its surrounding environment. Because to determine whether a pixel is a flame, you can't just look at how red it is, you also have to look at how much redder it is than its surrounding environment.
[0038] Example: Scene 1 - Dusk: The entire sky is red, and one pixel in the sky has a high Cr value, but its... It's also very high because the surrounding area is very bright, so Cr- This value may not be large, and the FSI score will not be high. This effectively suppresses the misjudgment of dusk in the sky as a fire.
[0039] Scene 2 - Flames in the Dark: A single flame pixel has a high Cr value, while the surrounding pixels... It's very low because the environment is dark, so Cr- This value will be very large, and the FSI score will be very high, thus enabling accurate identification of the flame.
[0040] Therefore, by using the above formula to calculate the FSI value of each pixel in the original image captured by the main camera, the FSI value of each pixel is obtained, thus forming an FSI feature map.
[0041] During this process, the Cr value of each pixel can also be extracted to obtain the Cr feature map corresponding to the original image.
[0042] In addition, texture analysis is performed on the original image, such as using the classic local binary pattern (LBI) to analyze the texture. During this process, the dynamic texture entropy of the flame (a concept introduced in this case, a mathematical index that quantifies the randomness, turbulence, and constant change of the flame texture) is introduced. Because the flame texture is random, turbulent, and constantly changing, its texture is complex (high entropy) and unstable; therefore, entropy features can also be combined to determine the fire situation. The dynamic texture entropy of the flame consists of two parts: Texture entropy: measures the complexity of textures in a single frame of an image. Dynamism: Measures the degree to which the above complexity changes over time. Only when a region's texture is both complex (high entropy) and drastically changing (large rate of entropy change) will its dynamic texture entropy score be high, thus being identified as a suspected flame region.
[0043] For texture entropy, the LBP (Local Binary Pattern) algorithm is used for calculation. LBP is an operator used to describe local texture features of an image, and it has significant advantages such as rotation invariance and grayscale invariance. For the window region corresponding to each pixel, the LBP value of each pixel in the window region can be calculated through the following LBP value calculation process: 1) Select a pixel in the image as the center.
[0044] 2) Take its surrounding 3x3 neighborhood (8 pixels).
[0045] 3) Use the gray value of the center pixel as the threshold.
[0046] 4) Compare the gray values of the surrounding 8 pixels with the threshold: those greater than or equal to the threshold are recorded as 1, and those less than the threshold are recorded as 0.
[0047] 5) Arrange these 8 0s or 1s in a fixed order (such as clockwise from the top left corner) to form an 8-bit binary number.
[0048] 6) Convert this 8-bit binary number to decimal (0-255). This decimal number is the LBP value of the center pixel.
[0049] After obtaining the LBP value of each pixel in the window region, an LBP histogram is further generated. The LBP histogram is a statistical distribution of the LBP values of all pixels in the original image.
[0050] In the LBP histogram: x-axis: LBP value (from 0 to 255, a total of 256 possible values).
[0051] The vertical axis represents the number of pixels with that LBP value.
[0052] As can be seen, in the LBP histogram, each LBP value corresponds to a bin, and the i-th bin is used to store the total number of all pixels in the original image with an LBP value of i.
[0053] Subsequently, the texture entropy value of the window region is calculated based on the LBP histogram. The calculation formula is as follows:
[0054] in, It is the probability of the i-th bin in the LBP histogram, and its calculation formula is:
[0055] It is evident that all The sum of all must be 1. This constitutes a probability distribution. In other words, the formula for calculating texture entropy above calculates the entropy of this probability distribution of the LBP histogram.
[0056] Example: If a region has a very uniform texture (such as a solid-color wall), then the LBP values of all pixels will be almost identical, and the histogram will be highly concentrated in one bin. The distribution is extremely uneven, resulting in a very low calculated entropy value.
[0057] If the texture of a region is very complex and chaotic (such as flames or dense tree canopy), various LBP values will appear, and the histogram will be relatively flat. The distribution is relatively uniform, and the calculated entropy value is very high.
[0058] As can be seen, a window region corresponds to a texture entropy value, which is the texture feature of the center pixel of the window region, representing the complexity of the micro-environment in which the center pixel is located.
[0059] Therefore, after traversing all pixels of the original image, the texture entropy value of each pixel can be obtained, thus obtaining the static texture feature map.
[0060] The above process can be simplified into the following steps: a) Initialize a static texture feature map with the same image resolution as the original image; b) Use a detection window of a preset size (e.g., 5x5) to traverse each pixel in the image in a sliding window manner; c) For each traversal, align the center of the detection window with the current pixel and calculate the region texture entropy of the image area covered by the window; d) The calculated region texture entropy value is used as the texture feature value of the current pixel and assigned to the pixel point at the corresponding coordinates in the static texture feature map; e) Repeat steps b) to d) until all pixels of the image have been traversed, and finally a complete static texture feature map is generated.
[0061] Using the above-described method for generating static texture feature maps, a static texture feature map of one frame of an image can be generated. To quantify the dynamics of texture changes, the following analysis is also required: Subtracting the static texture feature map of the current frame from the static texture feature map of the previous frame, i.e., performing a subtraction operation for each pixel, takes pixel i with coordinates (x, y) as an example, its texture change rate is:
[0062] in, This represents the texture entropy value of pixel i in the current frame. This represents the texture entropy value of pixel i in the previous frame.
[0063] Therefore, the texture change rate of each pixel can be calculated, and thus a texture change feature map can be obtained.
[0064] In summary, step S21 yields three feature maps: the FSI feature map, the Cr feature map, and the texture variation feature map.
[0065] Step S22: Based on the FSI feature map, Cr feature map, and texture variation feature map, a comprehensive feature map that integrates multimodal information and adaptive weights is obtained. The fusion process can be achieved by superimposing images according to set weights, i.e.:
[0066] in, For comprehensive feature maps; , , The weights are preset and can be set as needed; the sum of the three is 1. For FSI feature maps, For Cr characteristic map, This is a texture variation feature map.
[0067] In other variations, to improve the rationality of weight configuration, this invention also provides a dynamic adaptive weight scheme. The core of this scheme is to use a pre-built scene weight prediction model (which can be trained using a lightweight scene-aware attention network. The task is to dynamically generate three weight maps based on the current scene's illumination and complexity. During training, the model learns to reduce the brightness-sensitive FSI_map in dimly lit scenes). The weight of ) is increased to improve the chroma-sensitive Cr_map ( The weight of Entropy_map is adjusted; in complex backgrounds (such as leaves and bushes), the weight of Entropy_map is reduced. (Weights are assigned to avoid false alarms). In application, simply inputting the brightness channel of the original image into the model will output three weights. In short, these weights are dynamically generated by the scene weight prediction model: taking the image brightness channel as input, a self-supervised, lightweight attention network outputs weight maps corresponding to the FSI feature map, Cr feature map, and texture change feature map; the network training objective is to maximize the response of the fused comprehensive feature map in the real fire area and minimize the response in the background area, using binary cross-entropy or mean squared error as the loss function. More detailed, the model training process is as follows: ① Core training concept: self-supervised learning and task-driven learning The model's task is not "classification" but "weight generation," therefore it cannot be trained using traditional classification labels. A more advanced training method, more in line with the nature of its task, will be used: self-supervised learning.
[0068] The core idea is not to directly tell the network "what weights should be in what scenarios", but to give it a final goal and let it learn how to adjust the weights to best achieve that goal.
[0069] The ultimate goal is to make the integrated feature map (Integrated_Feature_Map) respond as strongly as possible to the real fire area and as weakly as possible to the background area.
[0070] ② Training samples Training samples are the "textbook" for model learning. In order to train a robust and generalizable model, the training samples must be diverse and have high-quality annotations.
[0071] The original data is a large-scale, diverse video dataset, including: Positive samples: Real fire videos in various scenarios (indoor fires, forest fires, candles, lighters, etc.).
[0072] Negative samples: Video clips that are highly likely to cause false alarms.
[0073] Color interference: Dusk / sunrise, red vehicles, red warning lights, red clothing.
[0074] Texture interference: swaying leaves, ripples on the water, fluttering flags.
[0075] Dynamic light source interference: car headlights, flashlights, screen flickering.
[0076] Complex scenarios: smoke, rain, snow, camera shake.
[0077] The samples are preprocessed and labeled. For each frame of the training video, the following processing is required: Input data: Extract the brightness channel Y of the image and use it as input to the model.
[0078] Generate intermediate feature maps: independently calculate Cr_map, FSI_map, and Entropy_map for this frame.
[0079] Generating "Guidance Labels": This is the most crucial step. Instead of creating labels for the weight maps W_cr, W_fsi, and W_entropy, we create an ideal target map for the final fusion result, which can be called the "Ideal Fusion Feature Map" (Ground_Truth_Integrated_Map). The creation method is as follows: The real fire area in each frame is accurately marked manually or through algorithms (Ground Truth Mask).
[0080] Create a zero-filled map of the same size as the original map as Ground_Truth_Integrated_Map.
[0081] At the pixel location corresponding to the actual fire area, assign a value of 1 (or a high value, such as 255).
[0082] In the background area, the value remains 0.
[0083] This label means: tell the model, "No matter how you adjust the weights, your final fusion result should be as close as possible to this image—the fire area is bright, and the rest is dark." ③ Model training process With the "textbook" (training samples) and the "standard answer" (Ground_Truth_Integrated_Map), training can begin.
[0084] The loss function is a benchmark for measuring how poorly the network performs. The goal is to make the Integrated Feature Map generated by the network as close as possible to the Ground Truth Integrated Map.
[0085] Definition: Using binary cross-entropy loss or mean squared error loss. Here, we take BCE as an example: Loss=BCE(Integrated_Feature_Map,Ground_Truth_Integrated_Map) This loss function penalizes behaviors that produce low predictions in the fire zone and high predictions in the background zone.
[0086] The specific training process is as follows, which is a standard deep learning training loop: Initialization: Randomly initialize the weight parameters of the "Dynamic Attention Network".
[0087] Forward propagation: Take a batch of data (containing multiple frames of images and their labels) from the training set.
[0088] For each frame: a. Input the Y channel into the network to obtain the three weight maps W_cr, W_fsi, and W_entropy predicted by the network.
[0089] b. Fuse these three weight maps with the pre-calculated Cr_map, FSI_map, and Entropy_map to obtain the Integrated_Feature_Map predicted by the network.
[0090] Calculate the loss: The network-predicted Integrated_Feature_Map is compared with the previously created Ground_Truth_Integrated_Map to calculate the Loss value.
[0091] Backpropagation and optimization: Use optimizers (such as Adam, SGD) to calculate gradients based on the loss value.
[0092] This gradient tells the network how to adjust its parameters (weights) to minimize the loss in the next prediction.
[0093] Update the network parameters.
[0094] Iteration: Repeat the above steps thousands of times until the network's loss value drops to a very low level and performs well on an independent validation set.
[0095] Step S23: Perform threshold segmentation and morphological operations on the comprehensive feature map to obtain preliminary candidate fire areas: The composite feature map is binarized by iterating through each pixel. Pixels with values less than or equal to a set threshold are set to 0 (black, representing the background), and pixels with values greater than the threshold are set to 255 (white, representing the foreground), resulting in a binarized image. This transforms the composite feature map into a clear black-and-white image, better distinguishing areas that might be flames from those that are definitely not.
[0096] Binary images may contain noise, such as isolated white pixels or holes (small black dots inside flames). To improve the accuracy of candidate fire areas, morphological processing techniques are first used to denoise the binary image, resulting in a cleaner and more complete binary image.
[0097] Next, the connected components of the binary graph are identified. This can be done using the Two-pass algorithm or a Union-Find-based algorithm.
[0098] Once the connected components are obtained, the properties of each connected component can be calculated, including: Bounding box: The smallest rectangle (x_min, y_min, x_max, y_max) that contains the connected region.
[0099] Pixel area: The total number of pixels contained in this area.
[0100] Centroid coordinates: the geometric center of this region.
[0101] Not all connected components are qualified candidates. A noisy region of only a few pixels is not worth the subsequent complex spatiotemporal verification.
[0102] a) Spatiotemporal verification - Region growing verification Set a size threshold: Based on the camera resolution and the scene being monitored, set a minimum area threshold Min_Area (e.g., 50 pixels).
[0103] Filtering: Traverse all connected components and discard all regions with an area smaller than Min_Area.
[0104] Output: The remaining connected components of a certain size represent the candidate fire areas for 3D verification. For each candidate area, the system has synchronously recorded its position in the main camera image and can find the corresponding area in the image after correction by the secondary camera. Each area can be represented by its bounding box.
[0105] For each candidate fire area, a spatiotemporal verification is performed (the core of the verification is that real fires are continuous, while interference such as reflections is usually instantaneous): a historical record buffer is established for each candidate fire area, which is used to store the state information (including area and bounding box) of the candidate fire area in the past N frames (e.g., N=5).
[0106] b. Next, perform persistence checks and growth trend checks on the buffer zone: Persistent checks: Check if there are M consecutive frames (e.g., M=3) in the buffer that contain the candidate fire area: Traverse the buffer and count the number of consecutive non-empty records. If the number of records is greater than or equal to M, then pass the persistence check. Because a reflective point that only appears in one frame will have a buffer history of {null, null, null, Rt, null}.
[0107] Growth trend check: To check whether the area of candidate fire zones in the buffer shows an overall increasing trend: extract the area of all non-empty records in the buffer to form an area sequence A=[A1, A2...Ak]. Calculate the linear fitting slope k of this sequence. If k>0, a growth trend is considered to exist; otherwise, no growth trend exists. This is because the area of a real fire, especially an early one, usually expands, while the area of a stable disturbance such as a red indicator light remains basically unchanged.
[0108] Quantifying growth factors: The results of the two tests mentioned above are quantified into a single growth factor G: If both checks are passed, then G = 1.2 (a reward is given); If only persistence is checked, then G=1.0 (neutral); Other cases: G=0.5 (a penalty is imposed).
[0109] c. Spatiotemporal verification - flicker frequency analysis: Verification core: Real flames have a unique, physically determined flicker frequency; Obtain the brightness Y change curve of each candidate fire area in the current frame over a past period of time.
[0110] Establish a time-series buffer of length L (e.g., corresponding to the data of the most recent second; if the frame rate is 25fps, then L=25). That is, obtain the brightness data of candidate regions in each frame within the 1 second preceding the current frame.
[0111] For each frame, calculate the average luminance value Y_avg of all pixels in the candidate region, and store Y_avg in the time series buffer to obtain a luminance signal sequence: Signal = [Y_avg_t, Y_avg_{t-1}, ..., Y_avg_{t-L+1}].
[0112] Transforming the luminance signal from the time domain (luminance changing over time) to the frequency domain (analyzing which frequencies the signal comprises): Performing a Fast Fourier Transform on the luminance signal sequence Signal yields a complex array, typically of interest in its amplitude spectrum or power spectrum. The power spectrum represents the energy intensity of each frequency component.
[0113] Next, the power spectrum is analyzed: Flame characteristic frequency range: According to research, the flicker energy of the flame is mainly concentrated in the low frequency range of 1Hz to 10Hz, that is: f_min = 1 Hz, f_max = 10 Hz; Calculate and obtain the flicker confidence score: In the power spectrum results of the FFT, find all data points corresponding to the frequency range from f_min to f_max. Flicker_Confidence = max(FFT_Power_Spectrum[f_{min}, f_{max}]) This formula means finding the power value (i.e., flicker confidence) of the frequency point with the strongest energy within the characteristic frequency range of the flame. The higher this value, the more the flicker behavior of the candidate region matches the physical characteristics of the flame.
[0114] In step S3, based on the correspondence between the candidate fire areas in the corrected images of the main and secondary cameras, stereo matching and three-dimensional spatial positioning are performed, and the stereo verification confidence factor is calculated.
[0115] In this step, for each candidate fire area output from step 2, stereo matching is performed using the secondary camera image to calculate its disparity and convert it into three-dimensional spatial coordinates. This step can calculate the actual distance to the fire point, estimate its physical size, and filter out interference that is difficult to eliminate with monocular vision, such as viewing angle and reflections. For example, a distant red roof may resemble a flame in a monocular image, but stereo vision reveals that it is extremely far away and huge, which does not conform to the characteristics of an initial fire, and thus it is excluded. Based on the quality of stereo matching and the degree of conformity between the three-dimensional size and the flame model, a stereo confidence factor (Stereo_Confidence) is calculated for each candidate area, with a value range of [0,1].
[0116] In step S4, based on the multimodal visual feature fusion result, spatiotemporal verification result, and stereo verification confidence factor, a comprehensive confidence score is calculated for each candidate fire area; candidate fire areas with a comprehensive confidence score greater than a set score threshold are designated as target suspected fire areas, and an alarm is triggered.
[0117] In this step, the final suspected fire area is determined based on the candidate fire areas: The stereo verification results are fused to calculate the final confidence score for each candidate region. The score consists of three parts: a comprehensive feature value based on the main camera image features, a spatiotemporal growth and scintillation factor, and a stereo verification confidence factor (Stereo_Confidence).
[0118] The formula for calculating the overall confidence score is:
[0119] in, is the value of the i-th pixel in the candidate fire area on the comprehensive feature map, N is the total number of pixels in the area, G is the growth factor quantified according to the growth trend check result, Flicker_Confidence is the flicker confidence, and Stereo_Confidence is the stereo verification confidence factor, whose value range is [0,1], and is jointly determined by the confidence of stereo matching and the degree of three-dimensional size conformity.
[0120] Specifically, candidate fire areas with a comprehensive confidence score greater than a set score threshold can be designated as target suspected fire areas and an alarm can be triggered.
[0121] It is understood that, in the embodiments of the present invention, the traditional methods of static thresholds and fixed weights are abandoned, and the problem of false alarms under complex lighting and backgrounds is solved by using dynamic attention networks.
[0122] At the same time, it creatively defines the "flame saturation index" and "flame dynamic texture entropy", which can better depict the essential physical characteristics of flames than a single color or texture.
[0123] In addition, a spatiotemporal dimension was introduced. By analyzing region growth trends and flicker frequency, the detection was elevated from a "two-dimensional image space" to a "three-dimensional spatiotemporal space," effectively filtering out transient interference. This is a fundamental improvement over single-frame detection.
[0124] Furthermore, a stereo vision dimension was introduced. Through the collaboration of the main and secondary cameras, three-dimensional localization and physical size estimation of the fire point were achieved, elevating the alarm from two-dimensional image coordinates to real spatial coordinates, fundamentally solving the problem that monocular systems cannot determine distance and true scale.
[0125] like Figure 3 The diagram shown illustrates a structural schematic of an embodiment of a wildfire early warning system 1 based on multi-view visual spatiotemporal features provided by the present invention. (In conjunction with...) Figure 4 As shown, in this embodiment, the system 1 includes at least: The multi-view image acquisition module 10 includes a main camera and a secondary camera with fixed spatial positions, used to synchronously acquire images of the monitored scene; Image processing and computing module 11 is communicatively connected to the multi-view image acquisition module and is used to receive and process images; The image processing and calculation module 11 is configured to perform: The images from the main and secondary cameras are subjected to stereo correction and matching to calculate depth information; Execute based on main camera image Figure 1 The described method includes multimodal feature extraction, fusion, candidate region extraction, and spatiotemporal verification steps; The candidate fire areas are subjected to 3D matching and 3D localization, and the 3D verification confidence factor is calculated. The comprehensive confidence score is calculated based on the multimodal feature fusion results, spatiotemporal verification results, and stereo verification results, and an early warning signal is generated.
[0126] like Figure 4 As shown, in a specific example, the image processing calculation module 11 further includes: The feature extraction and fusion unit 110 is used to calculate the flame saturation index (FSI) feature map, red chromaticity (Cr) feature map, and dynamic texture change feature map of the input image; call the pre-trained weighted adaptive model to generate dynamic weights for the three feature maps; and perform weighted summation on the three feature maps according to the dynamic weights to output a comprehensive feature map. The spatiotemporal verification unit 111 is used to maintain a historical state buffer for each candidate fire area to check the persistence and area growth trend of the area; and to establish a brightness time sequence buffer to obtain the flicker frequency confidence of the area by performing frequency domain analysis on the sequence. The stereo verification and positioning unit 112 is used to perform stereo matching and three-dimensional spatial coordinate calculation for each candidate fire area in combination with the secondary camera image, and generate a stereo verification confidence factor based on the matching quality and three-dimensional size conformity.
[0127] Furthermore, the weight adaptive model 11 is a scene-aware attention network trained based on self-supervised learning; the training objective of the network is to maximize the distinguishability of the resulting comprehensive feature map between the real fire area and the background area after fusing the dynamic weights output by the network with the pre-computed FSI feature map, Cr feature map and texture change feature map.
[0128] Furthermore, the image processing computing module 11 is deployed in one of the following locations: a local processor integrated into the main camera or the secondary camera, an independent local processing center communicating with the camera, or a remote server in the cloud.
[0129] For more details, please refer to and combine with the above. Figure 1 The description of that will not be repeated here.
[0130] Implementing this embodiment has the following beneficial effects: This invention provides a wildfire early warning method and system based on multi-view visual spatiotemporal features. This invention employs stereo matching and 3D positioning using primary and secondary cameras. Unlike traditional monocular systems, it calculates the true distance and 3D size of the fire point through dual-camera parallax, effectively overcoming interference from two-dimensional similar objects such as clouds and fog, achieving precise spatial positioning of the fire, and providing crucial information for rescue and dispatch.
[0131] This invention employs an adaptive multi-feature fusion mechanism. It innovatively integrates flame saturation index, chromaticity features, and dynamic texture entropy, and utilizes an attention network to dynamically adjust the weights of each feature. This enables the system to adapt to changes in illumination and complex backgrounds, significantly improving detection robustness under different environments and reducing false alarms.
[0132] This invention employs a spatiotemporal dual verification mechanism. By introducing regional growth trend analysis and flame flashing frequency analysis, the persistence and dynamic characteristics of the fire are verified from a temporal perspective. This effectively filters out instantaneous interference such as flashes and reflections, improving the sensitivity of identifying real early-stage fires.
[0133] This invention employs a combined 3D and temporal confidence assessment. By integrating 3D verification confidence, growth factor, and scintillation confidence, a final alarm decision is generated, forming a multi-layered filtering approach that combines spatial, temporal, and feature-based methods. This approach ensures a high detection rate while achieving an extremely low false alarm rate.
[0134] These technologies work together to enable the system to detect fires earlier and more accurately, and to output the three-dimensional location and spread trend of the fire, truly achieving a leap from "two-dimensional alarm" to "three-dimensional situational awareness"; it can reduce false alarms, improve the accuracy of distance and size judgment, and enhance the timeliness and accuracy of wildfire early warning.
[0135] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0136] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0137] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.
Claims
1. A wildfire early warning method based on multi-view visual spatiotemporal features, characterized in that, Includes the following steps: Images of the monitored scene are simultaneously acquired by a main camera and a secondary camera deployed at the same horizontal level, and the images are then subjected to stereoscopic correction processing. Multimodal visual features are extracted from the main camera image and adaptively fused to generate a comprehensive feature map; The comprehensive feature map is processed to obtain preliminary candidate fire areas; the candidate fire areas are then subjected to two-dimensional spatiotemporal verification, which includes analyzing the persistence, growth trend, and flicker frequency of the areas. Based on the correspondence between the candidate fire areas in the corrected images of the main and secondary cameras, stereo matching and three-dimensional spatial positioning are performed, and the stereo verification confidence factor is calculated. Based on the multimodal visual feature fusion results, spatiotemporal verification results, and stereo verification confidence factor, calculate the comprehensive confidence score for each candidate fire area; Candidate fire areas with a comprehensive confidence score greater than a set score threshold are designated as target suspected fire areas, and an alarm is triggered.
2. The method according to claim 1, characterized in that, The extraction and adaptive fusion of multimodal visual features specifically includes: Extract the luminance component Y, red chromaticity component Cr, and blue chromaticity component Cb of each pixel from the main camera image; The flame saturation index (FSI) of each pixel is calculated based on the luminance component Y, the red chromaticity component Cr, and the blue chromaticity component Cb, forming an FSI feature map. The FSI value of each pixel is calculated using the following formula: in, This represents the average brightness value within a local neighborhood centered on that pixel. Extract the Cr feature map corresponding to the main camera image; Calculate the dynamic texture features of the main camera image and generate a texture change feature map; The FSI feature map, Cr feature map, and texture change feature map are input into a pre-trained weight adaptive model to obtain their respective dynamic weight maps. The three feature maps are weighted and fused according to the dynamic weight map to generate the comprehensive feature map.
3. The method according to claim 2, characterized in that, The calculation of the dynamic texture features of the main camera image and the generation of a texture change feature map specifically includes: For each frame of the main camera image, the texture entropy of the local window where each pixel is located is calculated by a sliding window to generate a static texture feature map, wherein the texture entropy is calculated based on the local binary mode LBP histogram. Subtract the texture entropy value of the corresponding pixel in the static texture feature map of the current frame from the static texture feature map of the previous frame, and take the absolute value to obtain the texture change rate of each pixel, thereby generating the texture change feature map. The calculation process of the LBP value includes: Select a pixel in the image as the center; Take 8 pixels from its surrounding 3x3 neighborhood; The gray value of the center pixel is used as the threshold and compared with the gray values of the surrounding 8 pixels to generate an 8-bit binary number, which is then converted into a decimal number as the LBP value.
4. The method according to claim 3, characterized in that, In the aforementioned spatiotemporal verification, the flicker frequency analysis specifically includes: The average brightness value of each candidate fire area over the past L consecutive frames is obtained to form a brightness time sequence signal; The power spectrum is obtained by performing a fast Fourier transform on the brightness time-series signal. The maximum power value of the power spectrum within a preset flame characteristic frequency range is calculated as the scintillation confidence level of that region, wherein the flame characteristic frequency range is from 1 Hz to 10 Hz.
5. The method according to claim 4, characterized in that, The Stereo Confidence factor is determined by the confidence of the stereo matching and the degree to which the three-dimensional dimensions of the candidate region conform to the flame model, and its value range is [0,1]. The formula for calculating the overall confidence score is as follows: in, Let be the value of the i-th pixel in the candidate fire area on the comprehensive feature map, N be the total number of pixels in the area, G be the growth factor quantified according to the growth trend check results, Flicker_Confidence be the flicker confidence, and Stereo_Confidence be the stereo verification confidence factor.
6. The method according to any one of claims 1 to 5, characterized in that, The weights are dynamically generated through a scene weight prediction model: Using the image brightness channel as input, a lightweight attention network trained under self-supervised guidance outputs the weight maps corresponding to the FSI feature map, Cr feature map, and texture change feature map. The goal of network training is to maximize the response of the fused integrated feature map in the real fire area and minimize the response in the background area. The loss function is either binary cross-entropy or mean squared error.
7. A wildfire early warning system based on multi-view visual spatiotemporal features, characterized in that, include: The multi-view image acquisition module includes a main camera and a secondary camera with fixed spatial positions, used to synchronously acquire images of the monitored scene; The image processing and computing module is communicatively connected to the multi-view image acquisition module and is used to receive and process images; The image processing and calculation module is configured to perform: The images from the main and secondary cameras are subjected to stereo correction and matching to calculate depth information; Perform the multimodal feature extraction, fusion, candidate region extraction, and spatiotemporal verification steps as described in any one of claims 1 to 6 based on the main camera image; The candidate fire areas are subjected to 3D matching and 3D localization, and the 3D verification confidence factor is calculated. The comprehensive confidence score is calculated based on the multimodal feature fusion results, spatiotemporal verification results, and stereo verification results, and an early warning signal is generated.
8. The system according to claim 7, characterized in that, The image processing calculation module further includes: The feature extraction and fusion unit is used to calculate the flame saturation index (FSI) feature map, red chromaticity (Cr) feature map, and dynamic texture change feature map of the input image; call the pre-trained weighted adaptive model to generate dynamic weights for the three feature maps; and perform weighted summation on the three feature maps according to the dynamic weights to output a comprehensive feature map. The spatiotemporal verification unit is used to maintain a historical state buffer for each candidate fire area to check the persistence and area growth trend of the area; a brightness time sequence buffer is established, and the flicker frequency confidence of the area is obtained by performing frequency domain analysis on the sequence; The stereo verification and positioning unit is used to perform stereo matching and three-dimensional spatial coordinate calculation for each candidate fire area by combining the secondary camera image, and to generate a stereo verification confidence factor based on the matching quality and three-dimensional size conformity.
9. The system according to claim 8, characterized in that, The weight adaptive model is a scene-aware attention network trained based on self-supervised learning. The training objective of the network is to maximize the distinguishability of the resulting comprehensive feature map between the real fire area and the background area after fusing the dynamic weights output by the network with the pre-computed FSI feature map, Cr feature map, and texture change feature map.
10. The system according to any one of claims 7 to 9, characterized in that, The image processing computing module is deployed in one of the following locations: a local processor integrated into the main camera or the secondary camera, an independent local processing center communicating with the camera, or a remote server in the cloud.