A smoke fire detection method based on feature matching and multi-frame time sequence fusion

Abstract

The present application relates to a smoke fire detection method based on feature matching and multi-frame time sequence fusion, which is suitable for highway fire detection scene. The method first screens out foreground change pictures based on the steady-state entropy change of video frame pixels. Then, the color and texture features of smoke fire are matched using the fast flooding algorithm to screen out candidate pictures from the foreground pictures. Next, a cache queue is established to store the candidate pictures and their timestamps, and the multi-frame time sequence image data in the queue is sent to the detection model based on the preset trigger condition. Finally, the multi-frame time sequence fusion smoke fire detection algorithm model extracts features and analyzes the time sequence fusion of the input multi-frame images, and outputs the final fire and smoke detection results. The present application realizes efficient detection through low-power cameras, reduces false alarm rate, improves complex environment monitoring distance, and has lower power cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video picture processing, in particular to a smoke fire detection method based on representation matching and multi-frame time sequence fusion. BACKGROUND

[0002] In the field of expressways, fire hazard problems are particularly prominent due to the characteristics of expressways, such as being closed, having large traffic flow, and vehicles traveling at high speed. Once accidents such as vehicle self-ignition, cargo fire, or surrounding forest fire occur, they often cause serious traffic congestion and a large number of casualties in a short time. Therefore, it is of great practical significance to quickly and accurately detect smoke and fire on expressways.

[0003] At present, there is a multi-modal perception fire detection method based on a Transformer time sequence feature fusion model (application publication number: CN119380157A). This method relates to the field of fire safety monitoring technology and detects fire by combining a video, a multi-band infrared flame detector, and an ultraviolet sensor. The flame detector mainly consists of an image sensor, four infrared sensors, and an ultraviolet sensor. The image sensor is mainly used to sense wavelength information in the range of 400nm-800nm, the infrared sensor is sensitive to thermal radiation, and the ultraviolet sensor can detect ultraviolet light generated by fire. This method can adapt to the situation of missing part of the sensor data in actual data, and the multi-modal data fusion module designed by the method can effectively extract features from sensor data and image data, improve the generalization ability of the model, and capture the correlation between different sensor data and the deep feature information of the time sequence pattern specific to fire from long-distance context information, realizing adaptive fusion of historical data and current data.

[0004] However, the entire process of this invention requires high computing power of the deployment device and relies on a large number of sensors. In contrast, this invention only needs to rely on a camera, use a smoke fire representation matching algorithm to match images, and only use a multi-frame time sequence fusion detection model to infer and detect images with high matching degrees. This approach greatly reduces the data that the model needs to infer, thereby reducing the computing power requirement. Compared with existing technology, this method relies on fewer sensors, requires lower computing power of the deployment device, and has lower overall cost. SUMMARY

[0005] The present application provides a low-computing-power camera-based multi-frame time sequence fusion smoke fire detection method, which can quickly identify smoke and fire on expressways and adapt to various complex environments of mountainous expressways. This method greatly reduces the false alarm rate of fire alarms and improves the monitoring distance of smoke and fire.

[0006] The specific steps include:

[0007] S1. Screening out the picture of foreground change based on the change of pixel steady state entropy in the video frame. The change of pixel steady state entropy determination comprises: S101. Pixel level mutation detection: by establishing a dynamic background model, the current frame is compared with the background model pixel by pixel, and when the absolute value of the gray difference of the pixel point exceeds the change threshold ε, the pixel point is marked as a potential foreground pixel point.

[0008] S102. Pixel level time sequence steady state tracking and noise filtering: the potential foreground pixel points obtained in step S101 are time sequence tracked for N consecutive frames, and the change amplitude of each pixel point between adjacent frames is calculated. When the absolute value of the change amplitude of a pixel point in the consecutive N frames is less than or equal to the stability threshold ω, it is determined that the pixel point enters a steady state. All the pixel points entering the steady state are determined as a detection region ROI, wherein N is a predetermined positive integer.

[0009] S103. Local area entropy value calculation and foreground confirmation: taking the mutation occurrence time as the starting point, the pixel gray distribution in the detection region ROI is calculated after a delay of M frames, and the entropy value change P of the gray distribution relative to the corresponding region gray distribution of the initial background model. When P > K × , it is determined that there is a real moving foreground target. Wherein, M is a predetermined positive integer, is the area of the detection region, and K is a predetermined entropy change threshold.

[0010] S2. Based on the dynamic foreground picture screened out in S1, the color and texture features of smoke and fire in the picture are matched using a fast flooding algorithm, candidate pictures with smoke and fire features are screened out, and other pictures are eliminated.

[0011] S3. A cache queue is established to cache the candidate pictures screened out in step S2 and the corresponding timestamp information; based on a preset trigger condition, the multi-frame time sequence image data cached in the cache queue is sent to step S4 for processing.

[0012] S4. The multi-frame time sequence image data transmitted from S3 is input into a multi-frame time sequence fusion smoke and fire detection model; the model performs feature extraction on the input continuous multi-frame pictures through a multi-layer feature extraction network; the extracted multi-frame features are stored in a short-term memory network structure for operation; finally, the fire and smoke detection results of the picture are output.

[0013] Preferably, the type of foreground change is limited to four categories of vehicles, smoke, flames and pedestrians.

[0014] Preferably, the change of pixel steady state entropy determination in step S1 further comprises:

[0015] a. Pixel difference calculation: let I(x, y, t) be the gray value of the pixel point (x, y) in the current t frame, and BG(x, y, t-1) be the gray value of the corresponding position of the background model in the t-1 frame. The gray difference F(x, y, t) is expressed as: F(x, y, t) = I(x, y, t) - BG(x, y, t-1).

[0016] b. Mutation determination: set the change threshold ε. When the absolute value of the pixel point F(x, y, t) is greater than ε, it is determined that the pixel point has a gray mutation triggered by a moving target, and is marked as a potential foreground region.

[0017] c. Time series stability analysis: track the time series of the mutation pixel point for 10 consecutive frames, and calculate the pixel change amplitude V(x, y, t+k) between adjacent frames, which is expressed as:

[0018] V(x, y, t+k) = I(x, y, t+k) - I(x, y, t+k-1), where k = 1, 2, 3,..., 9.

[0019] d. Stability determination: set the stability threshold ω. When all V(x, y, t+k) in the 10 consecutive frames satisfy |V(x, y, t+k)|≤ω, it is determined that the pixel point enters a stable state, and it is added to the detection region .

[0020] e. Entropy calculation: take the mutation occurrence time t as the starting point, and calculate the entropy change of the pixel point relative to the initial background corresponding pixel point in the t+100 frames when the pixel point reaches a stable state again. The formula is:

[0021]

[0022] where p(t+100) is the ROI entropy value distribution function at t+100 frames, t represents the time, ROI represents the detection region, and e represents the base of natural logarithm, e takes the value 2.71828.

[0023] f. Foreground determination: when >1.43* , it is determined that there is a moving foreground target, where represents the area of the detection region.

[0024] Preferably, S2 comprises:

[0025] S21. Initialize the data structure: build a dictionary structure Fire_square storing the smoke fire area, the key of the dictionary is the area index identifier i, and the value is the area Si of the detection area corresponding to i; create a queue Fire_neighbord storing the pixel points (x, y) that need to be traversed in the fast flooding process; initialize a two-dimensional array Visited_nodes with the same size as the picture, the value of Visited_nodes[x][y] in the array is 0 or 1; set the pointers left_pos and right_pos, left_pos points to the head of the queue Fire_neighbord, and right_pos points to the tail of the queue.

[0026] S22. Seed node screening: flood the detection area image pixel points in the order of first row and then column from the top left corner of the detection area; select the pixel points that meet R(x, y) > G(x, y) > B(x, y), 255 > R(x, y) > 93, 78 < G(x, y) < 126, and Visited_nodes[x][y] = 0 as seed nodes, mark the coordinates of the selected seed nodes as (x, y), add them to the Fire_neighbord queue, and mark Visited_nodes[x][y] as 1, where R(x, y), G(x, y), and B(x, y) represent the red, green, and blue channel values of the image pixel point (x, y), respectively.

[0027] S23. Double-thread fast flooding diffusion: by setting the pointer left_pos to point to the head of the queue Fire_neighbord and the pointer right_pos to point to the tail of the queue Fire_neighbord, a double-thread concurrent access method is used to traverse the queue Fire_neighbord in both directions.

[0028] S24. Effective area determination: traverse the Fire_square dictionary, remove the areas with Si≤50 corresponding to the detection area, and when the number of remaining areas is >2, it is determined that the image contains target areas meeting the smoke and fire characteristics.

[0029] Preferably, in the bidirectional traversal process:

[0030] When the left_pos pointer accesses from the head to the tail and the right_pos pointer accesses from the tail to the head, the traversal operation is continuously executed.

[0031] When the left_pos and right_pos pointers point to the same address, it is determined that the fast flooding traversal of the current seed node is completed.​​​

[0032] Further, during the traversal process, for each current seed node pixel coordinate pointed to by the pointer ( ), a 4-neighborhood traversal method is used to check its neighborhood pixel points (x_neg, y_neg). When the neighborhood pixel points (x_neg, y_neg) meet the following conditions: color feature R(x_neg, y_neg) > G(x_neg, y_neg) > B(x_neg, y_neg), 93 < R(x_neg, y_neg) < 255, 78 < G(x_neg, y_neg) < 126, red channel pixel gradient Grd(x_neg, y_neg) > 9.2, and Visited_nodes[x_neg, y_neg] = 0; where the calculation formula for the red channel pixel gradient Grd(x_neg, y_neg) is:

[0033] .

[0034] Add the neighborhood pixel points (x_neg, y_neg) to the end of the temporary queue Fire_neighbord_tmp and mark Visited_nodes[x_neg, y_neg] as 1, and move the current pointer to the next seed node in the queue until the left_pos and right_pos pointers point to the same address.

[0035] Further, when the left_pos and right_pos pointers point to the same address: when the Fire_neighbord_tmp queue is not empty, cut the Fire_neighbord_tmp queue to the Fire_neighbord queue, reset the left_pos and right_pos pointers to point to the head and tail of the Fire_neighbord queue respectively, and then perform the S23 traversal.

[0036] Further, when the left_pos and right_pos pointers point to the same address: when the Fire_neighbord_tmp queue is empty, store the area Si of the current region in the dictionary structure Fire_square, end the flooding process of the current seed node, and continue to execute S22 to select a new seed node.

[0037] Preferably, the cache queue in step S3 includes an image queue and a timestamp queue. The image queue is configured to cache candidate pictures, and the timestamp queue is configured to cache the acquisition timestamps corresponding to the candidate pictures.

[0038] The preset trigger conditions include:

[0039] When the number of images cached in the cache queue reaches a preset capacity threshold, the sending operation is triggered. After the sending operation is triggered, the cache queue is emptied, and the candidate pictures screened out in the S2 step and the corresponding timestamp information are continuously cached.

[0040] When the number of images cached in the cache queue does not reach the preset capacity threshold, and the interval between the timestamp of the latest candidate picture in the cache queue and the current time exceeds a preset time threshold, the cache queue is emptied, and the candidate pictures screened out in the S2 step and the corresponding timestamp information are continuously cached.

[0041] Preferably, the operation in the short-term memory network structure includes the following parallel operations:

[0042] (a) multi-layer fusion of stored features;

[0043] (b) performing associated cross fusion of features based on a time sequence dimension.

[0044] Advantages of the present application

[0045] Compared with the prior art, the present application only needs to rely on an image sensor, first uses a smoke fire feature matching algorithm to match the image, only uses a multi-frame time sequence fusion detection model to infer and detect the image with high matching degree, and does not need to infer and detect data without smoke fire, greatly reduces the data that needs to be inferred by the model, thereby reducing the algorithm requirement, the model training data set is all smoke fire data in a high-speed scene, further improves the fitting ability of the model, the detection rate in the test set is as high as 96.8%, and real-time detection can be achieved at the camera end, and only a camera with low algorithm needs to be deployed in the area to be detected in engineering deployment. The method not only has low cost but also is convenient to deploy. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 It is a schematic diagram of the overall process of the present application;

[0048] Figure 2 It is a schematic diagram of R channel color threshold statistics;

[0049] Figure 3 It is a schematic diagram of G channel color threshold statistics;

[0050] Figure 4 A multi-frame time sequence fusion smoke fire detection algorithm model schematic diagram;

[0051] Figure 5 A YOLOV11 training index and precision schematic diagram;

[0052] Figure 6 A multi-frame time sequence fusion smoke fire detection algorithm model index and precision schematic diagram;

[0053] Figure 7 A YOLOV11-M detection result schematic diagram;

[0054] Figure 8 A multi-frame time sequence fusion model detection result schematic diagram. DETAILED DESCRIPTION

[0055] The detection algorithm of the embodiment is deployed on the camera side or the background server side, is deployed on the chip with an algorithm power of 2T or more that is self-provided by the camera to detect and run in real time, and the camera resolution is 2K or more; the server with a display card is deployed, and the video stream is pushed to the server side to run detection based on the TensorRT environment.

[0056] As shown in Figure 1 , the embodiment mainly includes four steps:

[0057] S1. Screening the picture with foreground changes based on the change of pixel steady state entropy in the video frame.

[0058] The picture with foreground changes is screened in the video frame. In actual high-speed scenes, only four types of targets, i.e., cars, smoke, flames and people, can cause foreground changes. In the embodiment, a formula for the change of pixel steady state entropy is designed to screen the picture with foreground changes. The picture without foreground changes does not enter the next step of calculation.

[0059] Through a large number of statistics, it is found that the situation with foreground changes only accounts for about 67% in actual projects. The screening of the change of pixel steady state entropy greatly reduces the calculation amount of hardware. The calculation of the pixel steady state entropy is simple and can screen the non-static foreground data while reducing the calculation amount of CPU.

[0060] The calculation of the pixel steady state entropy includes three steps:

[0061] (1) The embodiment adopts the inter-frame difference method for foreground detection. Specifically, the current frame is compared with the background model in terms of pixel point gray value by establishing a dynamic background model. Let I(x, y, t) be the gray value of the current t frame at the (x, y) position, BG(x, y, t-1) be the gray value of the corresponding position of the t-1 frame background model, and the difference F(x, y, t) between the two reflects the instantaneous change of the current frame.

[0062] The amplitude of the foreground pixel change of the current pixel point in the detection area is calculated, which is used for foreground mutation detection and dynamic region positioning:

[0063]

[0064] Wherein, represents the gray value of the pixel point with coordinates (x, y) in the t-frame video sequence, is the gray value of the pixel point with coordinates (x, y) in the background established by the t-1 frame video sequence, and when | ( is the allowable change degree, in the embodiment =14), it is determined that there is a moving target passing through the background pixel point region, resulting in a large amplitude fluctuation of the gray value of the pixel point.

[0065] (2) After detecting the instantaneous mutation of the gray value of the pixel point (x, y), the stable change amplitude of the gray value between adjacent frames in the time sequence of the pixel point is counted in a continuous time, which is used for stable tracking and noise filtering:

[0066]

[0067] When there is ≤ ( is the stability degree, in the embodiment =3), it is recorded that from the t+k frame, when there are |≤ , k = 1, 2,..., j-1, it is determined that the pixel point starts to enter a stable state.

[0068] It should be noted that random noise (such as camera shaking, car light flickering) is usually accompanied by unordered frame difference fluctuations (absolute value is large or positive and negative alternates) after mutation, which cannot meet the stable condition that the difference values of continuous multiple frames are less than the threshold. The smoke or flame pixel points related to real fire (such as smoke diffusion and flame edge spread) often show gradual changes in gray value (small and stable adjacent frame difference) at the micro level, so the stable state can be used to effectively screen out candidate regions with high confidence, providing reliable input for subsequent entropy analysis.

[0069] In a real scene, in order not to miss detection as much as possible, we do not process the pixel points that do not pass the stable state determination, and only keep the pixel points that pass the stable state determination.

[0070] For the pixel points that pass the stable state determination, the pixel point is added to the detection area , that is The set of stable pixel points is mutated.

[0071] (3) Calculate the change of pixel entropy after the current steady state change, which is used for entropy value calculation and foreground confirmation:

[0072] When the pixel point reaches a steady state from a fluctuating state, the entropy value change of the pixel point gray value in the t+100 frame relative to the initial background corresponding pixel point is calculated. Entropy is mainly used to reflect the order degree of the system. The greater the entropy value, the more chaotic the system. In this embodiment, a pixel entropy change formula is designed to calculate whether there is a dynamic foreground:

[0073]

[0074] Wherein, p(t+100) is the ROI entropy value distribution function at t+100 frame, t represents the time, ROI represents the detection area, e represents the base number of natural logarithm, and e takes the value 2.71828.

[0075] Finally, the entropy calculation method is used to calculate the change of the foreground in a short time for the static change pixel point. >1.43 Then it is determined that there is a moving foreground target. Wherein represents the area of the detection area.

[0076] It should be noted that the initial background is the frame before the pixel gray value mutation in the dynamic background model.

[0077] S2. Based on the dynamic foreground selected in S1, the fast flooding algorithm is used to match the color and texture features of smoke and fire in the picture, and the pictures with smoke and fire features are selected and other pictures are eliminated.

[0078] The main steps include:

[0079] S21. Initialize the data structure: build a dictionary structure Fire_square to store the smoke and fire area, the dictionary structure area index identifier i, the area is recorded as , and the value is the area of the area (i represents the i-th area in the picture); create a Fire_neighbord queue, which is used to store the pixel point coordinates (x, y) that need to be traversed in the fast flooding process; set the size of the Visited_nodes array to be consistent with the size of the picture, which is used to identify whether the pixel point has been visited in the fast flooding process. The value in Visited_nodes[x][y] is 1, which represents that it has been visited, and 0 represents that it has not been visited; the left_pos and right_pos two pointers point to the head and tail of the Fire_neighbord queue respectively, which are used for double-threaded fast flooding.

[0080] S22. Seed Node Filtering: First, traverse the image, flooding from the top left corner of the detection area in row-first, then column-first order. During the flooding traversal, select the seed nodes that flood quickly. ), where the seed node must satisfy R( )>G( B( ),255>R( )>93,78<G( ) < 126 and Visited_nodes[ ][ ]=0 (meaning the seed node has not been visited), after selecting the seed node, the seed node ( Add the visited nodes to the Fire_neighbord queue and set the visited nodes to the Fire_neighbord queue. ][ The value is set to 1, where RGB represents the three color channels in the image. These three color thresholds were obtained through histogram statistics of 10,000 high-speed fire data points. The statistical results for the R and G channels are as follows: Figure 2 and Figure 3 As shown.

[0081] S23. Dual-threaded fast flooding: The Fire_neighbord queue is subjected to dual-threaded fast flooding. Dual-threaded fast flooding mainly relies on two pointers, left_pos and right_pos, which point to the front and rear of the Fire_neighbord queue, respectively. The left_pos pointer accesses from the front to the rear, while the right_pos pointer accesses from the rear to the front. The fast flooding algorithm relies on these two pointers for bidirectional concurrent access until the left_pos and right_pos pointers point to the same address, which means that the traversal of the queue is over.

[0082] During each pointer thread traversal, a 4-neighborhood traversal is used to expand the queue for each seed node. Specifically, the pixel coordinates of the current seed node are retrieved from the address pointed to by the pointer. ), check each neighborhood ( Does it satisfy color feature R( )>G( B( ),255>R( )>93,78<G( ) < 126 and its R channel pixel gradient >9.2, of which The calculation method is as follows:

[0083] .

[0084] Visited_nodes[ ][ ] is 0, then add ( ) to the end of Fire_neighbord_tmp and set Visited_nodes[ ][ ] to 1, move the current pointer to the next seed node in the queue until the left_pos and right_pos point to the same address. When the Fire_neighbord_tmp queue is not empty, cut the Fire_neighbord_tmp queue to Fire_neighbord and point the left_pos and right_pos to the head and tail of the Fire_neighbord queue respectively and continue to execute S23; when the Fire_neighbord_tmp queue is empty, it means that the fast flooding of the seed node is over, store the area of the region to the Fire_square dictionary. Continue to S22 to select a new seed node.

[0085] S24. Effective area determination: after the flooding of all seed nodes is over, traverse the Fire_square dictionary to eliminate the regions in Fire_square with area <=50 . After the final elimination, if the number of remaining regions is >2, it is considered that the picture has a region that meets the color and texture characteristics of the fire.

[0086] The double-thread fast flooding algorithm has an absolute advantage in traversal matching speed compared to the commonly used BFS (Breadth-First Search), especially in the Cuda environment. After comparison test, its traversal speed is 2.3 times that of the commonly used BFS search algorithm.

[0087] S3. Maintain the cache queue of time sequence multi-frame, establish global shared queue Queue img and Queue time, capacity is 5, Queue img stores 5 frames of pictures and Queue time stores corresponding time stamp. When S2 detects the picture that may exist smoke fire, the picture is stored into the cache queue, when the cache queue is full, 5 time sequence pictures are transmitted to S4 for detection, whether the detection is real fire or not, the data in the queue is emptied; When the number of pictures in the queue is not full, and the difference T between the time stamp of the latest picture in the queue and the current time is greater than 3 minutes, the previous cache frame in the queue is emptied, and the latest picture and the latest time stamp are cached in the queue, so that it can be ensured that the queue stores continuous pictures with smoke fire characteristics in a short time.

[0088] S4. Smoke and fire detection algorithm in the detection picture is fused by using multi-frame time sequence fusion:

[0089] When the cache queue Queue img in S3 is full, 5 pictures in Queue img are sent as input to the multi-frame time sequence fusion smoke and fire detection algorithm model for inference detection, wherein the overall architecture of the model is as shown in Figure 4 .

[0090] The algorithm model first uses a multi-layer feature extraction network to extract features from the input continuous multi-frame pictures, stores the extracted multi-frame continuous features in a short-term memory network structure, simultaneously performs multi-layer fusion on the short-term reliable memory features, and performs cross fusion based on the time sequence dimension, and finally fuses the feature map to output the fire and smoke in the detection picture. The biggest advantage of the algorithm model over the common smoke and fire monitoring algorithm is that the algorithm inputs multiple continuous pictures, which can accurately identify smoke and fire based on multi-frame changes, and at the same time, the time sequence network model and the latest YOLOV11-M model are compared in the same data set. The comparison test index and precision are as shown in Figure 5 and Figure 6 . The curve graph of mAP50 of the multi-frame time sequence fusion smoke and fire detection algorithm model in the training process is as shown in Figure 6 . The data set has a total of 3.2W, and a total of 200 rounds are trained. The maximum value of mAP50 in the training process is the 195th round, and the maximum value is 96.77%; compared with YOLOV11-M trained for 200 rounds on the same data set (as Figure 5As shown in the figure), the maximum value of 87.51% was obtained in the 149th round. The detection model of multi-frame time sequence fusion has an advantage of 9.26% in accuracy. And the multi-frame time sequence fusion algorithm model can analyze multiple pictures and extract dynamic changing features. Especially in the tunnel scene, the brake light is easy to cause misidentification of the single-frame analysis model, while the multi-frame time sequence fusion model can analyze multiple changing results and extract the features of the jumping flame, not limited to the color features of the single smoke flame.

[0091] The effect schematic diagram is as shown in Figure 7 and Figure 8 .

[0092] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A smoke fire detection method based on representation matching and multi-frame temporal fusion, characterized in that, include: S1. Filter out images with foreground changes based on the changes in pixel steady-state entropy in video frames; The determination of the change in pixel steady-state entropy includes: S101. Pixel-level mutation detection: By establishing a dynamic background model, the gray values ​​of the current frame and the background model are compared pixel by pixel. When the absolute value of the gray value difference of a pixel exceeds the change threshold ε, the pixel is marked as a potential foreground pixel. S102. Pixel-level temporal steady-state tracking and noise filtering: Perform temporal tracking on the potential foreground pixels obtained in step S101 for N consecutive frames, and calculate the change amplitude of each pixel between adjacent frames; when the absolute value of the change amplitude of a pixel in N consecutive frames is lower than or equal to the stability threshold ω, the pixel is determined to have entered a steady state; the set of all pixels that have entered a steady state is determined as the detection region ROI; where N is a preset positive integer; S103. Local Region Entropy Calculation and Foreground Confirmation: Starting from the moment of the mutation, after a delay of M frames, calculate the entropy change P of the pixel grayscale distribution within the detection region (ROI) relative to the grayscale distribution of the corresponding region in the initial background model; when P > K × If M is a preset positive integer, then it is determined that a real moving foreground target exists; where M is a preset positive integer. Where K is the area of ​​the detection region, and K is the preset entropy change threshold. S2. Based on the dynamic foreground images selected in S1, use the fast flooding algorithm to match the color and texture features of smoke and fire in the images, select candidate images with smoke and fire features, and remove other images; S3. Establish a cache queue to cache the candidate images selected in step S2 and their corresponding timestamp information; based on a preset trigger condition, send the multi-frame time-series image data cached in the cache queue to step S4 for processing; S4. Input the multi-frame temporal image data transmitted from S3 into the multi-frame temporal fusion smoke and fire detection model; the model performs feature extraction on the input continuous multi-frame images through a multi-layer feature extraction network; the extracted multi-frame features are stored in a short-term memory network structure for operation; finally, the fire and smoke detection results of the images are output.

2. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The types of foreground changes are limited to four categories: vehicles, smoke, flames, and pedestrians.

3. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The determination of pixel steady-state entropy change in step S1 further includes: a. Pixel difference calculation: Let I(x,y,t) be the gray value of pixel (x,y) in the current frame t, and BG(x,y,t-1) be the gray value of the corresponding position of the background model in frame t-1. The gray value difference F(x,y,t) is expressed by the formula: F(x,y,t)=I(x,y,t)-BG(x,y,t-1); b. Abrupt change determination: Set a change threshold ε. When the absolute value of pixel F(x,y,t) is greater than ε, it is determined that there is a gray-scale abrupt change triggered by a moving target and it is marked as a potential foreground region. c. Temporal stability analysis: Track the time series of pixels with abrupt changes for 10 consecutive frames, and calculate the magnitude of pixel change V(x,y,t+k) between adjacent frames. The formula is as follows: V(x,y,t+k)=I(x,y,t+k)-I(x,y,t+k-1), where k=1,2,3...,9; d. Steady-state determination: Set a stability threshold ω. When all V(x,y,t+k) of a pixel satisfy |V(x,y,t+k)|≤ω within 10 consecutive frames, the pixel is determined to have entered a steady state and is added to the detection region. ; e. Entropy Calculation: Starting from the moment t when the mutation occurs, calculate the change in the grayscale value of the pixel relative to the corresponding pixel in the initial background in frames t+100 after the pixel returns to a steady state from a fluctuating state. The formula is: Among them, p(t + 100) is the ROI entropy value distribution function at the (t + 100)-th frame, t represents the time, ROI represents the detection area, e represents the base of the natural logarithm, and the value of e is 2.71828; f. Foreground assessment: When >1.43* Then it is determined that a moving foreground target has appeared, where This represents the area of ​​the detection zone.

4. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The S2 includes: S21. Initialize the data structure: Construct a dictionary structure Fire_square for storing the smoke and fire areas. The key of the dictionary is the area index identifier i, and the value is the area Si of the i-corresponding detection area; Create a queue Fire_neighbord, which stores the pixel points (x, y) to be traversed during the fast flooding process; Initialize a two-dimensional array Visited_nodes with the same size as the image. The value of Visited_nodes[x][y] in the array is 0 or 1; Set pointers left_pos and right_pos, where left_pos points to the head of the queue Fire_neighbord, and right_pos points to the tail of the queue. S22. Seed node screening: Flood-traverse the pixels of the detection area image in the order of rows first and then columns from the upper left corner of the detection area; Select pixels that satisfy R(x,y)>G(x,y)>B(x,y), 255>R(x,y)>93, 78<G(x,y)<126 and Visited_nodes[x][y]=0 as seed nodes; Record the coordinates of the selected seed nodes as ( ) and add them to the Fire_neighbord queue, and mark Visited_nodes as 1; where R(x,y), G(x,y), and B(x,y) represent the red, green, and blue channel values of the image pixel (x,y) respectively;​ S23. Dual-thread fast flooding diffusion: By setting the pointer left_pos to point to the head of the queue Fire_neighbord and the pointer right_pos to point to the tail of the queue Fire_neighbord, traverse the queue Fire_neighbord bidirectionally in a dual-thread concurrent access manner; 5. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 4, characterized in that, S24. Determine the valid area: Traverse the Fire_square dictionary. After excluding the areas where the area Si of the i-corresponding detection area is ≤ 50, when the number of remaining areas > 2, it is determined that there are target areas in the image that conform to the characteristics of smoke and fire. During the bidirectional traversal process: When the left_pos pointer accesses from the head to the tail of the queue and the right_pos pointer accesses from the tail to the head of the queue, continuously execute the traversal operation; 6. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 5, characterized in that, During the traversal, for each pixel coordinate of the current seed node pointed to by the pointer ( The neighboring pixels (x_neg, y_neg) are checked using a 4-neighbor traversal method. ； When the left_pos and right_pos pointers point to the same address, it is determined that the fast flooding traversal of the current seed node ends. When the neighborhood pixel point (x_neg, y_neg) satisfies the following conditions: color feature R(x_neg, y_neg) > G(x_neg, y_neg) > B(x_neg, y_neg), 93 < R(x_neg, y_neg) < 255, 78 < G(x_neg, y_neg) < 126, the red channel pixel gradient Grd(x_neg, y_neg) > 9.2, and Visited_nodes[x_neg, y_neg] = 0; The calculation formula for the red channel pixel gradient Grd(x_neg, y_neg) is: Add the neighborhood pixel point (x_neg, y_neg) to the tail of the temporary queue Fire_neighbord_tmp and mark Visited_nodes[x_neg, y_neg] as 1, and move the current pointer to the next seed node in the queue until the left_pos and right_pos pointers point to the same address.

7. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 6, characterized in that, When the left_pos and right_pos pointers point to the same address: If the Fire_neighbord_tmp queue is not empty, then cut the Fire_neighbord_tmp queue to the Fire_neighbord queue, reset the left_pos and right_pos pointers to point to the head and tail of the Fire_neighbord queue respectively, and then perform the S23 traversal.

8. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 7, characterized in that, When the left_pos and right_pos pointers point to the same address: If the Fire_neighbord_tmp queue is empty, the area Si of the current region is stored in the dictionary structure Fire_square; End the flooding process of the current seed node; Continue executing S22 to select a new seed node.

9. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The cache queue in step S3 includes an image queue and a timestamp queue; the image queue is configured to cache candidate images, and the timestamp queue is configured to cache the acquisition timestamps corresponding to the candidate images. The preset triggering conditions include: When the number of images cached in the cache queue reaches a preset capacity threshold, the sending operation is triggered; after the sending operation is triggered, the cache queue is cleared, and the candidate images selected in step S2 and their corresponding timestamp information are cached again. If the number of images cached in the cache queue does not reach the preset capacity threshold, and the interval between the timestamp of the latest candidate image in the cache queue and the current time exceeds the preset time threshold, then the cache queue is cleared, and the candidate images and their corresponding timestamp information selected in step S2 are cached again.

10. The smoke fire detection method based on representation matching and multi-frame temporal fusion according to claim 1, characterized in that, The operations performed in the short-term memory network structure include the following parallel operations: (a) Multi-layer fusion of stored features; (b) Perform correlation and cross-fusion of features based on the time-series dimension.

Images