Smoke fire detection method based on representation matching and multi-frame time sequence fusion

Abstract

The invention relates to a smog fire detection method based on representation matching and multi-frame time sequence fusion, which is suitable for a highway fire detection scene. According to the method, firstly, foreground change pictures are screened out based on steady-state entropy change of video frame pixels; then, matching the color and texture features of the smoke fire disaster by using a fast flooding algorithm, and screening out candidate pictures from the foreground pictures; then, establishing a cache queue to store the candidate pictures and timestamps thereof, and sending multi-frame time sequence image data in the queue to a detection model based on a preset triggering condition; and finally, performing feature extraction and time sequence fusion analysis on the input multi-frame image through a multi-frame time sequence fusion smoke fire detection algorithm model, and outputting a final fire and smoke detection result. According to the invention, efficient detection is realized through the low-computing-power camera, the false alarm rate is reduced, the complex environment monitoring distance is increased, and the computing power cost is lower.

Description

Technical Field

[0001] This invention relates to the field of video and image processing technology, specifically to a smoke and fire detection method based on representation matching and multi-frame temporal fusion. Background Technology

[0002] Highways are particularly vulnerable to fire hazards due to their enclosed nature, high traffic volume, and high vehicle speeds. Incidents such as vehicle fires, cargo fires, or nearby forest fires can quickly cause severe traffic congestion and result in numerous casualties. Therefore, rapid and accurate detection of smoke and fires on highways is of paramount importance.

[0003] Currently, there is a multimodal sensing fire detection method based on a Transformer temporal feature fusion model (application publication number: CN119380157A). This method relates to the field of fire safety monitoring technology and uses a flame detector that combines video, multi-band infrared, and ultraviolet sensing technologies for fire detection. The detector mainly consists of an image sensor, four infrared sensors, and one ultraviolet sensor. The image sensor is mainly used to sense wavelength information in the 400nm-800nm ​​range, the infrared sensors are sensitive to thermal radiation, and the ultraviolet sensor can detect the ultraviolet light generated by the flame. This method can adapt to situations where some sensor data is missing in actual data, and its designed multimodal data fusion module can effectively extract features from sensor data and image data, improving the model's generalization ability. It can also capture the correlation between different sensor data and the deep feature information of fire-specific time series patterns from long-distance contextual information, achieving adaptive fusion of historical and current data.

[0004] However, the entire process of this invention requires significant computing power from the deployment equipment and relies on a large number of sensors. In contrast, this invention only requires a camera, uses a smoke and fire characterization matching algorithm to match images, and only uses a multi-frame temporal fusion detection model for inference detection on images with high matching degrees. This approach greatly reduces the amount of data the model needs to infer, thereby reducing the computing power requirements. Compared to existing technologies, this method relies on fewer sensors, requires less computing power from the deployment equipment, and has a lower overall cost in engineering applications. Summary of the Invention

[0005] This invention provides a smoke and fire detection method based on multi-frame temporal fusion using low-computing-power cameras. This method can quickly identify smoke and fire on highways and can adapt to various complex environments on mountainous highways. It greatly reduces the false alarm rate of fire alarms and increases the detection distance of smoke and fire.

[0006] The specific steps include:

[0007] S1. Filter out images with foreground changes based on the changes in pixel steady-state entropy in video frames. The determination of changes in pixel steady-state entropy includes: (1) Foreground mutation detection and dynamic region localization: By establishing a dynamic background model, the gray values ​​of the current frame and the background model are compared pixel by pixel, the gray value difference is calculated, and the gray value mutation triggered by a moving target is determined based on the set change threshold. (2) Steady-state tracking and noise filtering: Multi-frame temporal tracking is performed on the mutated pixels, the pixel change amplitude between adjacent frames is calculated, the stability threshold is set to determine whether it has entered a steady state, and the steady-state pixels are added to the detection area. (3) Entropy calculation and foreground confirmation: Starting from the mutation time, the entropy change of the pixel gray value distribution in the detection area relative to the initial background model after it reaches a steady state again is calculated, and the existence of a moving foreground target is determined based on the entropy change.

[0008] S2. Based on the dynamic foreground images selected in S1, a fast flooding algorithm is used to match the color and texture features of smoke and fire in the images, filter out candidate images with smoke and fire features, and remove other images.

[0009] S3. Establish a cache queue to cache the candidate images selected in step S2 and their corresponding timestamp information; based on preset trigger conditions, send the multi-frame time-series image data cached in the cache queue to step S4 for processing.

[0010] S4. Input the multi-frame temporal image data transmitted from S3 into the multi-frame temporal fusion smoke and fire detection model; the model performs feature extraction on the input continuous multi-frame images through a multi-layer feature extraction network; the extracted multi-frame features are stored in a short-term memory network structure for operation; finally, the fire and smoke detection results of the images are output.

[0011] Preferably, the types of foreground changes are limited to four categories: vehicles, smoke, flames, and pedestrians.

[0012] Preferably, the determination of the pixel steady-state entropy change in step S1 further includes:

[0013] a. Pixel difference calculation: Let I(x,y,t) be the gray value of pixel (x,y) in the current frame t, and BG(x,y,t-1) be the gray value of the corresponding position of the background model in frame t-1. The gray value difference between the two is expressed by the formula: F(x,y,t)=I(x,y,t)-BG(x,y,t-1).

[0014] b. Change determination: Set a change threshold ε. When the absolute value of pixel F(x,y,t) is greater than ε, it is determined that there is a gray-scale change triggered by a moving target and it is marked as a potential foreground region.

[0015] c. Temporal stability analysis: Track the time series of pixels with abrupt changes for 10 consecutive frames, and calculate the magnitude of pixel change V(x,y,t+k) between adjacent frames. The formula is as follows:

[0016] V(x,y,t+k)=I(x,y,t+k)-I(x,y,t+k-1), where k=1,2,3...,9.

[0017] d. Steady-state determination: Set a stability threshold ω. When all V(x,y,t+k) of a pixel satisfy |V(x,y,t+k)|≤ω within 10 consecutive frames, the pixel is determined to have entered a steady state and is added to the detection region. .

[0018] e. Entropy Calculation: Starting from the moment t when the mutation occurs, calculate the change in the grayscale value of the pixel relative to the corresponding pixel in the initial background in frames t+100 after the pixel returns to a steady state from a fluctuating state. The formula is:

[0019]

[0020] Where p(t+100) is the ROI entropy distribution function at frame t+100, t represents time, ROI represents the detection region, e represents the base of the natural logarithm, and e takes the value 2.71828.

[0021] f. Foreground assessment: When >1.43* Then it is determined that a moving foreground target has appeared, where This represents the area of ​​the detection zone.

[0022] Preferably, S2 includes:

[0023] S21. Initialize data structure: Construct a dictionary structure Fire_square to store smoke and fire areas. The key of the dictionary is the area index identifier i, and the value is the area Si of the detection area corresponding to i. Create a queue Fire_neighbord to store the pixels (x, y) that need to be traversed during the fast flooding process. Initialize a two-dimensional array Visited_nodes with the same size as the image. The value of Visited_nodes[x][y] in the array is 0 or 1. Set pointers left_pos and right_pos. The left_pos points to the head of the queue Fire_neighbord, and the right_pos points to the tail of the queue.

[0024] S22. Seed node screening: Flood-traverse the pixels of the detection area image in the order of row first and column second from the upper left corner of the detection area; select the pixels that satisfy R(x,y)>G(x,y)>B(x,y), 255>R(x,y)>93, 78<G(x,y)<126 and Visited_nodes[x][y]=0 as seed nodes, and record the coordinates of the selected seed nodes as ( , ) and add them to the Fire_neighbord queue, and mark Visited_nodes as 1, where R(x,y), G(x,y), and B(x,y) respectively represent the red, green, and blue channel values of the image pixel point (x,y).

[0025] S23. Dual-threaded fast flood diffusion: By setting the pointer left_pos to point to the head of the Fire_neighbord queue and the pointer right_pos to point to the tail of the Fire_neighbord queue, perform a two-way traversal of the Fire_neighbord queue using a dual-threaded concurrent access method.

[0026] S24. Effective area determination: Traverse the Fire_squre dictionary, and after removing the areas where the area Si of the corresponding detection area i ≤ 50, when the number of remaining areas > 2, it is determined that there is a target area in the image that conforms to the characteristics of smoke and fire.

[0027] Preferably, during the two-way traversal process:

[0028] When the left_pos pointer accesses from the head to the tail of the queue and the right_pos pointer accesses from the tail to the head of the queue, continuously execute the traversal operation.

[0029] When the left_pos and right_pos pointers point to the same address, it is determined that the fast flood traversal of the current seed node ends.

[0030] Furthermore, during the traversal process, for each pixel coordinate of the current seed node pointed to by the pointer ( , ​), the 4-neighborhood traversal method is adopted to check its neighborhood pixel points (x_neg, y_neg). When the neighborhood pixel points (x_neg, y_neg) meet the following conditions: color feature R(x_neg, y_neg) > G(x_neg, y_neg) > B(x_neg, y_neg), 93 < R(x_neg, y_neg) < 255, 78 < G(x_neg, y_neg) < 126, the red-channel pixel gradient Grd(x_neg, y_neg) > 9.2, and Visited_nodes[x_neg, y_neg] = 0; where the calculation formula of the red-channel pixel gradient Grd(x_neg, y_neg) is:

[0031] .

[0032] Add the neighborhood pixel points (x_neg, y_neg) to the end of the temporary queue Fire_neighbord_tmp and mark Visited_nodes[x_neg, y_neg] as 1, and move the current pointer to the next seed node in the queue until the left_pos and right_pos pointers point to the same address.

[0033] Furthermore, when the left_pos and right_pos pointers point to the same address: when the Fire_neighbord_tmp queue is not empty, cut the Fire_neighbord_tmp queue to the Fire_neighbord queue, reset the left_pos and right_pos pointers to point to the head and tail of the Fire_neighbord queue respectively, and then perform the S23 traversal.

[0034] Furthermore, when the left_pos and right_pos pointers point to the same address: when the Fire_neighbord_tmp queue is empty, store the area Si of the current region in the dictionary structure Fire_squre, end the flooding process of the current seed node, and continue to execute S22 to select a new seed node.

[0035] Preferably, the cache queue in step S3 includes an image queue and a timestamp queue. The image queue is configured to cache candidate pictures, and the timestamp queue is configured to cache the acquisition timestamps corresponding to the candidate pictures.

[0036] The preset trigger conditions include:

[0037] When the number of images cached in the cache queue reaches a preset capacity threshold, the sending operation is triggered. After triggering the sending operation, the cache queue is cleared, and the candidate images selected in step S2 and their corresponding timestamp information are cached again.

[0038] If the number of images cached in the cache queue does not reach the preset capacity threshold, and the interval between the timestamp of the latest candidate image in the cache queue and the current time exceeds the preset time threshold, then the cache queue is cleared, and the candidate images and their corresponding timestamp information selected in step S2 are cached again.

[0039] Preferably, the operation performed in the short-term memory network structure includes the following parallel operations:

[0040] (a) Multi-layer fusion of stored features;

[0041] (b) Perform correlation and cross-fusion of features based on the time-series dimension.

[0042] Beneficial effects of the present invention

[0043] Compared with existing technologies, this invention only relies on image sensors. First, it uses a smoke and fire characterization matching algorithm to match images. Only images with high matching degree are inferred and detected using a multi-frame temporal fusion detection model, without needing to infer and detect data that does not contain smoke and fire characteristics. This greatly reduces the amount of data that the model needs to infer, thereby reducing the computational power requirement. The training dataset for this model consists of smoke and fire data in high-speed scenes, which further improves the model's fitting ability. The detection rate in the test set is as high as 96.8%, and it can achieve real-time detection at the camera end. In terms of engineering deployment, it only requires deploying cameras with low computing power in the areas that need to be detected. This method is not only low-cost but also easy to deploy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0045] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the statistical values ​​of the color thresholds for the R channel.

[0047] Figure 3 This is a schematic diagram illustrating the statistical values ​​of the color thresholds for the G channel.

[0048] Figure 4 A schematic diagram of a multi-frame temporal fusion smoke fire detection algorithm model;

[0049] Figure 5 This is a diagram illustrating the training metrics and accuracy of YOLOv11.

[0050] Figure 6 A schematic diagram illustrating the metrics and accuracy of a multi-frame temporal fusion smoke and fire detection algorithm model;

[0051] Figure 7 This is a schematic diagram of the YOLOV11-M detection results;

[0052] Figure 8 This is a schematic diagram of the detection results of a multi-frame temporal fusion model. Detailed Implementation

[0053] In this embodiment, the detection algorithm is deployed on the camera side or the backend server. When deployed on the camera side, the camera has a built-in chip with a computing power of more than 2T and runs in real time. The camera resolution is above 2K. The server is equipped with a graphics card and pushes the video stream to the server side for detection based on the TensorRT environment.

[0054] like Figure 1 As shown, this embodiment mainly consists of four steps:

[0055] S1. Filter out images with foreground changes based on the changes in pixel steady-state entropy in video frames.

[0056] Images with foreground changes are filtered out from video frames. In actual high-speed scenes, only four types of targets cause foreground changes: vehicles, smoke, flames, and people. In this embodiment, a formula for pixel steady-state entropy change is designed to filter out images with foreground changes. Images without foreground changes are not further calculated.

[0057] Based on extensive statistical analysis, foreground variations occur in only about 67% of actual projects. By filtering through pixel steady-state entropy changes, the computational load on the hardware is greatly reduced. The pixel steady-state entropy calculation is simple and can filter out non-static foreground data with low CPU computational load.

[0058] The calculation of pixel steady-state entropy consists of three steps:

[0059] (1) In this embodiment, the inter-frame difference method is used for foreground detection. The specific method is as follows: by establishing a dynamic background model, the gray values ​​of the current frame and the background model are compared pixel by pixel. Let I(x,y,t) be the gray value of the current frame at position (x,y), and BG(x,y,t-1) be the gray value of the corresponding position of the background model in frame t-1. The difference between the two, F(x,y,t), reflects the instantaneous change of the current frame.

[0060] Calculate the magnitude of foreground pixel change at the current pixel within the detection area, used for foreground abrupt change detection and dynamic region localization:

[0061]

[0062] in, This represents the grayscale value of the pixel at coordinates (x, y) in the t-frame video sequence. For the background pixel with coordinates (x, y) established in the t-1 frame video sequence, when | |> ( To allow for variability, in this embodiment If the value is 14, it is determined that a moving target has passed through the background pixel area, causing a large fluctuation in the grayscale value of the pixel.

[0063] (2) After detecting an instantaneous change in the gray value of the pixel (x,y), the steady-state change amplitude of the gray value between adjacent frames in the time series of the pixel is statistically analyzed over a continuous period of time for steady-state tracking and noise filtering:

[0064]

[0065] When | ≤ ( For stability, in this embodiment =3), record from frame t+k, when in consecutive frames j (j=10 in this embodiment) there is | |≤ If k=1,2.......,j-1, then the pixel is determined to have entered a stable state.

[0066] It should be noted that random noise (such as camera shake or headlight flicker) is usually accompanied by disordered fluctuations in inter-frame differences (large absolute values ​​or alternating positive and negative values) after abrupt changes, which cannot satisfy the steady-state condition that the differences in multiple consecutive frames are all less than a threshold. However, smoke or flame pixels related to real fires (such as smoke diffusion or flame edge spread) often exhibit gradual changes in grayscale values ​​at the microscopic level (small and stable differences between adjacent frames). Therefore, steady-state determination can effectively screen out high-confidence candidate regions, providing reliable input for subsequent entropy analysis.

[0067] In real-world scenarios, to minimize missed detections, we do not process pixels that fail the steady-state determination and only retain pixels that pass the steady-state determination.

[0068] For pixels that pass the steady-state determination, add the pixel to the detection area. ,Right now It is the set of pixels that are stable during mutation.

[0069] (3) Calculate the pixel entropy change after the current steady-state change, which is used for entropy calculation and foreground confirmation:

[0070] Once the pixel reaches a steady state after fluctuating, the entropy change of the pixel's grayscale value relative to the initial background pixel is calculated in frame t+100. Entropy mainly reflects the degree of order in a system; the higher the entropy value, the more chaotic the system. In this embodiment, a formula for calculating the change in pixel entropy is designed to determine whether a dynamic foreground has appeared.

[0071]

[0072] Where p(t+100) is the ROI entropy distribution function at frame t+100, t represents time, ROI represents the detection region, e represents the base of the natural logarithm, and e takes the value 2.71828.

[0073] Finally, entropy calculation is used to calculate the changes in the foreground over a short period of time for statically changing pixels. >1.43* Then it is determined that a moving foreground target has appeared, where This represents the area of ​​the detection zone.

[0074] It should be noted that the initial background is the frame immediately preceding the sudden change in pixel grayscale values ​​in the dynamic background model.

[0075] S2. Based on the dynamic foreground selected in S1, use the fast flooding algorithm to match the color and texture features of smoke and fire in the images, filter out images with smoke and fire features, and remove other images.

[0076] Its main steps include:

[0077] S21. Initialize data structure: Construct a dictionary structure Fire_squre to store the smoke and fire areas. The dictionary structure has an area index identifier i, and the area is denoted as... , where value is the area of ​​the region. (i represents the i-th region in the image). Create a `Fire_neighbord` queue to store the coordinates (x, y) of the pixels to be traversed during the fast flooding process. Set the size of the `Visited_nodes` array to the same size as the image to indicate whether a pixel has been visited during the fast flooding process; a value of 1 in `Visited_nodes[x][y]` indicates it has been visited, and 0 indicates it has not been visited. The `left_pos` and `right_pos` pointers point to the front and rear of the `Fire_neighbord` queue, respectively, for use in a dual-threaded fast flooding process.

[0078] S22. Seed Node Filtering: First, the image is traversed in the order of flooding from the top left corner of the detection area, row by row. During the flooding traversal, seed nodes that are flooded quickly are selected. , ), where the seed node must satisfy R( , )>G ( , >B () , ),255>R( , >93,78<G ( , ) < 126 and Visited_nodes[ ][ ]=0 (meaning the seed node has not been visited), after selecting the seed node, the seed node ( , Add the visited nodes to the Fire_neighbord queue and set the visited nodes to the Fire_neighbord queue. ][ The value is set to 1, where RGB represents the three color channels in the image. These three color thresholds were obtained through histogram statistics of 10,000 high-speed fire data points. The statistical results for the R and G channels are as follows: Figure 2 and Figure 3 As shown.

[0079] S23. Dual-threaded fast flooding: The Fire_neighbord queue is subjected to dual-threaded fast flooding. Dual-threaded fast flooding mainly relies on two pointers, left_pos and right_pos, which point to the front and rear of the Fire_neighbord queue, respectively. The left_pos pointer accesses from the front to the rear, while the right_pos pointer accesses from the rear to the front. The fast flooding algorithm relies on these two pointers for bidirectional concurrent access until the left_pos and right_pos pointers point to the same address, which means that the traversal of the queue is over.

[0080] During each pointer thread traversal, a 4-neighborhood traversal is used to expand the queue for each seed node. Specifically, the pixel coordinates of the current seed node are retrieved from the address pointed to by the pointer. , ), check each neighborhood ( Does it satisfy color feature R( )>G( B( ),255>R( )>93,78<G( ) < 126 and its R channel pixel gradient >9.2, of which The calculation method is as follows:

[0081] .

[0082] Meanwhile Visited_nodes[ ][ If the condition is 0, then ( Add it to the tail of the Fire_neighbord_tmp queue and then add Visited_nodes[ ][ Set the pointer to 1, move the current pointer to the next seed node in the queue, until the left_pos and right_pos pointers point to the same address. If the Fire_neighbord_tmp queue is not empty, cut the Fire_neighbord_tmp queue to Fire_neighbord, and set the left_pos and right_pos pointers to the head and tail of the Fire_neighbord queue respectively, and continue executing S23; if the Fire_neighbord_tmp queue is empty, it means that the fast flooding of this seed node has ended, and the region is... area Stored in the Fire_squre dictionary. Proceed to S22 to select a new seed node.

[0083] S24. Valid Region Determination: After the flooding of all seed nodes is complete, traverse the Fire_squre dictionary and remove areas from Fire_squre. Areas <=50 After final elimination, if the number of remaining areas is greater than 2, then the image is considered to contain areas that match the color and texture characteristics of a fire.

[0084] This dual-threaded fast flooding algorithm has an absolute advantage in traversal matching speed compared to the commonly used BFS (Breadth-First Search), especially in the CUDA environment. Comparative tests show that its traversal speed is 2.3 times that of the commonly used BFS search algorithm.

[0085] S3. Maintain a time-series multi-frame cache queue. Establish globally shared queues Queue_img and Queue_time, each with a capacity of 5. Queue_img stores 5 frames of images, and Queue_time stores the corresponding timestamps. When S2 detects an image that may indicate a smoke or fire, it stores the image in the cache queue. When the cache queue is full, it sends 5 time-series images to S4 for detection. Regardless of whether a fire actually exists, the data in the queue is cleared. If the queue is not full and the difference T between the timestamp of the latest image in the queue and the current time is greater than 3 minutes, the previous cached frames in the queue are cleared, and the latest image and its latest timestamp are cached in the queue. This ensures that the queue contains only consecutive images that exhibit smoke or fire characteristics within a short period of time.

[0086] S4. Utilize a multi-frame temporal fusion smoke and fire detection algorithm to fuse the fire and smoke in the output detection image:

[0087] When the cache queue Queue_img in S3 is full, the 5 images in Queue_img will be fed as input into the multi-frame temporal fusion smoke and fire detection algorithm model for inference and detection. The overall architecture of this model is as follows: Figure 4 As shown.

[0088] The algorithm first uses a multi-layer feature extraction network to extract features from multiple consecutive frames of input images. The extracted features are stored in a short-term memory network structure. Simultaneously, multi-layer fusion of the short-term memory features is performed, and feature cross-fusion is performed based on temporal dimensions. Finally, the fused feature maps are output to detect fire and smoke in the images. The biggest advantage of this algorithm compared to common smoke and fire detection algorithms is that it takes multiple consecutive frames of images as input and can accurately identify smoke and fire based on multi-frame changes. Furthermore, ablation comparison experiments were conducted on this temporal network model and the latest YOLOv11-M model on the same dataset. The comparison metrics and accuracy are as follows: Figure 5 and Figure 6 As shown: The mAP50 curve of the multi-frame temporal fusion smoke and fire detection algorithm model during the training process is as follows. Figure 6 As shown, the dataset contains 32,000 images and was trained for 200 epochs. The maximum mAP50 value during training was reached in epoch 195, with a maximum value of 96.77%. In comparison, YOLOv11-M trained for 200 epochs on the same dataset (e.g., ...). Figure 5As shown in the figure, the maximum value of 87.51% was achieved in the 149th round. The multi-frame temporal fusion detection model showed a significant advantage, exceeding the model's accuracy by 9.26%. Furthermore, the multi-frame temporal fusion algorithm can analyze multiple images to extract dynamically changing features. Especially in tunnel scenes, brake lights can easily cause misidentification by single-frame analysis models, while the multi-frame temporal fusion model analyzes multiple changing results, extracting the characteristics of flame flickering, rather than being limited to the color features of a single smoke and flame image.

[0089] Its effect diagram is as follows Figure 7 and Figure 8 As shown.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smoke fire detection method based on representation matching and multi-frame temporal fusion, characterized in that, include: S1. Filter out images with foreground changes based on the changes in pixel steady-state entropy in video frames; The determination of the change in steady-state entropy of the pixels includes: (1) Foreground mutation detection and dynamic region localization: By establishing a dynamic background model, the gray values ​​of the current frame and the background model are compared pixel by pixel, the gray value difference is calculated, and the gray value mutation triggered by the moving target is determined based on the set change threshold; (2) Steady-state tracking and noise filtering: Multi-frame temporal tracking is performed on the mutated pixels, the pixel change amplitude between adjacent frames is calculated, the stability threshold is set to determine whether it has entered the steady state, and the steady-state pixels are added to the detection area; (3) Entropy calculation and foreground confirmation: Starting from the mutation time, the entropy change of the pixel gray distribution in the detection area relative to the initial background model after reaching the steady state again is calculated, and the existence of the moving foreground target is determined based on the entropy change; S2. Based on the dynamic foreground images selected in S1, use the fast flooding algorithm to match the color and texture features of smoke and fire in the images, select candidate images with smoke and fire features, and remove other images; S3. Establish a cache queue to cache the candidate images selected in step S2 and their corresponding timestamp information; based on a preset trigger condition, send the multi-frame time-series image data cached in the cache queue to step S4 for processing; S4. Input the multi-frame temporal image data transmitted from S3 into the multi-frame temporal fusion smoke and fire detection model; the model performs feature extraction on the input continuous multi-frame images through a multi-layer feature extraction network; the extracted multi-frame features are stored in a short-term memory network structure for operation; finally, the fire and smoke detection results of the images are output.

2. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The types of foreground changes are limited to four categories: vehicles, smoke, flames, and pedestrians.

3. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The determination of pixel steady-state entropy change in step S1 further includes: a. Pixel difference calculation: Let I(x,y,t) be the gray value of pixel (x,y) in the current frame t, and BG(x,y,t-1) be the gray value of the corresponding position of the background model in frame t-1. The gray value difference F(x,y,t) is expressed by the formula: F(x,y,t)=I(x,y,t)-BG(x,y,t-1); b. Abrupt change determination: Set a change threshold ε. When the absolute value of pixel F(x,y,t) is greater than ε, it is determined that there is a gray-scale abrupt change triggered by a moving target and it is marked as a potential foreground region. c. Temporal stability analysis: Track the time series of pixels with abrupt changes for 10 consecutive frames, and calculate the magnitude of pixel change V(x,y,t+k) between adjacent frames. The formula is as follows: V(x,y,t+k)=I(x,y,t+k)-I(x,y,t+k-1), where k=1,2,3...,9; d. Steady-state determination: Set a stability threshold ω. When all V(x,y,t+k) of a pixel satisfy |V(x,y,t+k)|≤ω within 10 consecutive frames, the pixel is determined to have entered a steady state and is added to the detection region. ; e. Entropy Calculation: Starting from the moment t when the mutation occurs, calculate the change in the grayscale value of the pixel relative to the corresponding pixel in the initial background in frames t+100 after the pixel returns to a steady state from a fluctuating state. The formula is: Where p(t+100) is the ROI entropy distribution function at frame t+100, t represents time, ROI represents the detection region, and e represents the base of the natural logarithm, with e taking the value 2.71828. f. Foreground assessment: When >1.43* Then it is determined that a moving foreground target has appeared, where This represents the area of ​​the detection zone.

4. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, S2 includes: S21. Initialize data structure: Construct a dictionary structure Fire_square to store smoke and fire areas. The key of the dictionary is the area index identifier i, and the value is the area Si of the detection area corresponding to i. Create a queue Fire_neighbord to store the pixels (x, y) that need to be traversed during the fast flooding process. Initialize a two-dimensional array Visited_nodes with the same size as the image. The value of Visited_nodes[x][y] in the array is 0 or 1. Set pointers left_pos and right_pos. The left_pos points to the head of the queue Fire_neighbord, and the right_pos points to the tail of the queue. S22. Seed node screening: Flood and traverse the pixels of the detection area image in the order of rows first and columns second from the upper left corner of the detection area; Select pixels that satisfy R(x,y)>G(x,y)>B(x,y), 255>R(x,y)>93, 78<G(x,y)<126 and Visited_nodes[x][y]=0 as seed nodes; Record the coordinates of the selected seed nodes as ( , ) and add them to the Fire_neighbord queue, and mark Visited_nodes as 1; where R(x,y), G(x,y), and B(x,y) represent the red, green, and blue channel values of the image pixel (x,y) respectively;​ S23. Dual-threaded fast flooding and diffusion: By setting the pointer left_pos to point to the head of the queue Fire_neighbord and the pointer right_pos to point to the tail of the queue Fire_neighbord, the queue Fire_neighbord is traversed bidirectionally using a dual-threaded concurrent access method. S24. Valid region determination: Traverse the Fire_squre dictionary, remove the regions whose area Si ≤ 50 corresponding to the detection region i, and when the number of remaining regions > 2, determine that the image has a target region that meets the characteristics of smoke and fire.

5. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 4, characterized in that, During the bidirectional traversal process: The traversal operation continues as the left_pos pointer moves from the front of the queue to the rear and the right_pos pointer moves from the rear of the queue to the front. When the left_pos and right_pos pointers point to the same address, the fast flooding traversal of the current seed node is considered to have ended.

6. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 5, characterized in that, During the traversal, for each pixel coordinate of the current seed node pointed to by the pointer ( , The neighboring pixels (x_neg, y_neg) are checked using a 4-neighbor traversal method. When the neighboring pixel (x_neg, y_neg) satisfies the following conditions: color feature R(x_neg, y_neg) > G(x_neg, y_neg) > B(x_neg, y_neg), 93 < R(x_neg, y_neg) < 255, 78 < G(x_neg, y_neg) < 126, the red channel pixel gradient Grd(x_neg, y_neg) > 9.2, and Visited_nodes[x_neg, y_neg] = 0; where the calculation formula for the red channel pixel gradient Grd(x_neg, y_neg) is: ; Add the neighboring pixels (x_neg, y_neg) to the tail of the temporary queue Fire_neighbord_tmp and mark Visited_nodes[x_neg, y_neg] as 1. Move the current pointer to the next seed node in the queue until the left_pos and right_pos pointers point to the same address.

7. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 6, characterized in that, When the left_pos and right_pos pointers point to the same address: If the Fire_neighbord_tmp queue is not empty, then cut the Fire_neighbord_tmp queue to the Fire_neighbord queue, reset the left_pos and right_pos pointers to point to the head and tail of the Fire_neighbord queue respectively, and then perform the S23 traversal.

8. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 7, characterized in that, When the left_pos and right_pos pointers point to the same address: If the Fire_neighbord_tmp queue is empty, the area Si of the current region is stored in the dictionary structure Fire_squre; End the flooding process of the current seed node; Continue executing S22 to select a new seed node.

9. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The cache queue in step S3 includes an image queue and a timestamp queue; the image queue is configured to cache candidate images, and the timestamp queue is configured to cache the acquisition timestamps corresponding to the candidate images. The preset triggering conditions include: When the number of images cached in the cache queue reaches a preset capacity threshold, the sending operation is triggered; after the sending operation is triggered, the cache queue is cleared, and the candidate images selected in step S2 and their corresponding timestamp information are cached again. If the number of images cached in the cache queue does not reach the preset capacity threshold, and the interval between the timestamp of the latest candidate image in the cache queue and the current time exceeds the preset time threshold, then the cache queue is cleared, and the candidate images and their corresponding timestamp information selected in step S2 are cached again.

10. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The operations performed in the short-term memory network structure include the following parallel operations: (a) Multi-layer fusion of stored features; (b) Perform correlation and cross-fusion of features based on the time-series dimension.

Images