Smoke and fire detection method and device based on video sequence

Abstract

The invention discloses a smoke and fire detection method and device based on a video sequence, and relates to the related field of computer vision, and the method comprises the steps: carrying out the target detection processing of a single-frame image in a video stream, and obtaining a detection frame set containing a smoke and fire potential region; for each potential smoke and fire area in the detection box set, processing continuous multi-frame images of the video stream based on a target tracking algorithm, and generating a plurality of smoke and fire trajectories representing the motion state of the potential smoke and fire area; and on the basis of the plurality of smoke and fire tracks, intercepting a corresponding continuous image sequence from the video stream to construct a video sequence sample, and processing the video sequence sample by using a pre-trained sequence classification model to obtain a judgment result of whether the video sequence sample is a real smoke and fire phenomenon. The technical problems that an existing smoke and fire detection method is high in false alarm rate and insufficient in dynamic smoke and fire characteristic recognition are solved, and the technical effects of reducing the false alarm rate and improving the accuracy of real smoke and fire dynamic behavior recognition are achieved.

Description

A method and apparatus for smoke detection based on video sequences Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method and apparatus for detecting smoke and fire based on video sequences. Background Technology

[0002] Smoke detection is crucial for safety in transportation and other scenarios. Timely and accurate smoke detection can effectively prevent casualties and property damage caused by fires. Currently, the main methods for solving this problem include traditional sensor detection methods and computer vision detection methods based on single-frame images. Traditional sensor detection methods are limited by their detection principles, resulting in problems such as response lag, susceptibility to environmental interference and false alarms, and an inability to comprehensively locate targets. Computer vision detection methods based on single-frame images, relying solely on information from a single frame, struggle to cope with complex backgrounds, changes in lighting, and other interference, leading to both false positives and false negatives.

[0003] Currently, existing smoke detection methods suffer from high false alarm rates and insufficient identification of dynamic smoke characteristics. Summary of the Invention

[0004] This application provides a video sequence-based method and apparatus for detecting fireworks. It employs target detection on single frames of a video stream to obtain a set of detection boxes for potential fireworks areas. A target tracking algorithm tracks each potential area across multiple consecutive frames, generating corresponding fireworks trajectories. Based on these trajectories, continuous image sequences are extracted to construct samples. These samples are then processed by a pre-trained sequence classification model to determine whether the phenomenon is a real fireworks event. This approach solves the technical problems of high false alarm rates and insufficient recognition of dynamic fireworks characteristics in existing fireworks detection methods, achieving the technical effect of reducing false alarm rates and improving the accuracy of recognizing real dynamic fireworks behavior.

[0005] This application provides a video sequence-based method for detecting fireworks, comprising: performing target detection processing on a single frame image in a video stream to obtain a set of detection boxes containing potential fireworks regions; for each potential fireworks region in the detection box set, processing multiple consecutive frames of the video stream based on a target tracking algorithm to generate multiple fireworks trajectories representing the motion state of the potential fireworks region; based on the multiple fireworks trajectories, extracting corresponding consecutive image sequences from the video stream to construct video sequence samples, and processing the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples represent real fireworks phenomena.

[0006] In a possible implementation, object detection processing is performed on a single frame image in the video stream to obtain a set of detection boxes containing potential fireworks regions. The following processing is then performed: an object detection model based on the YOLO architecture is used to perform object detection processing on the single frame image in the video stream, and the set of detection boxes containing the location of the bounding boxes of potential fireworks regions and their category confidence is output. The backbone network of the object detection model uses a C3K2 module integrating CSPNet and depthwise separable convolutions for feature extraction, performs multi-scale feature fusion through SPPF layers, uses a PAN structure as the neck network to enhance feature aggregation, introduces the CBAM attention mechanism in the feature fusion stage, uses a decoupled head design to separate classification and regression tasks, and applies an Anchor-Free mechanism to directly predict bounding boxes.

[0007] In a possible implementation, for each potential firework region in the detection box set, the video stream is processed using a target tracking algorithm across multiple consecutive frames to generate multiple firework trajectories representing the motion state of the potential firework region. The following processing is performed: for the detection box set in the current frame, it is divided into a high-confidence detection box set and a low-confidence detection box set based on a confidence threshold; the positions of multiple firework trajectories from the previous frame in the current frame are predicted using Kalman filtering to obtain a trajectory prediction box set; the trajectory prediction box set is matched in two stages with the high-confidence detection box set and the low-confidence detection box set to output the multiple firework trajectories after processing in the current frame.

[0008] In a possible implementation, a two-stage matching process is performed between the trajectory prediction box set, the high-confidence detection box set, and the low-confidence detection box set. Multiple fireworks trajectories are output after the current frame processing is completed. The following steps are then performed: the intersection-union ratio (IUU) of the trajectory prediction box set and the high-confidence detection box set is calculated; the Hungarian algorithm is used for the first round of matching; successfully matched trajectories and their corresponding detection boxes are associated; the trajectory status is updated to obtain a set of matched trajectories; and unmatched trajectories and unmatched high-confidence detection boxes are added to the unmatched trajectory set and the unmatched high-confidence detection box set, respectively. A set of boxes is generated; for each unmatched high-confidence detection box in the set of unmatched high-confidence detection boxes, a new trajectory is created to obtain a new trajectory set; the intersection-union ratio (IUGR) of the unmatched trajectory set and the set of low-confidence detection boxes is calculated, and a second round of matching is performed using the Hungarian algorithm. The successfully matched trajectories and their corresponding detection boxes are associated, the trajectory status is updated, and they are added to the set of matched trajectories; unmatched trajectories and detection boxes are discarded; all trajectories in the set of matched trajectories and the new trajectory set are output as multiple fireworks trajectories after the current frame is processed, for continuous tracking processing of the next frame of the video stream.

[0009] In a possible implementation, based on the multiple fireworks trajectories, corresponding continuous image sequences are extracted from the video stream to construct video sequence samples. These video sequence samples are then processed using a pre-trained sequence classification model to determine whether they represent real fireworks phenomena. The following processing is performed: based on the temporal range and spatial location covered by each fireworks trajectory, corresponding continuous image frames are extracted from the video stream to construct video sequence samples; the video sequence samples are input into a sequence classification model based on a 3D convolutional neural network for processing, and a determination result is output to determine whether the corresponding trajectory of the video sequence sample represents a real fireworks phenomenon; wherein, the 3D convolutional neural network uses 3D convolutional kernels to simultaneously extract the spatial and temporal dimensions of the video sequence samples to fuse spatiotemporal information and model the dynamic characteristics of fireworks.

[0010] In a possible implementation, the following processing is performed: the sequence classification model is trained using a video sequence dataset containing positive samples of real fireworks phenomena and negative samples of easily confused interference, and the ratio of positive to negative samples is balanced during training.

[0011] In one possible implementation, the following processing is performed: the video stream originates from a traffic monitoring scenario.

[0012] This application also provides a video sequence-based smoke detection device, comprising: a target detection processing module for performing target detection processing on a single frame image in a video stream to obtain a set of detection boxes containing potential smoke areas; a target tracking module for processing multiple consecutive frames of the video stream for each potential smoke area in the detection box set based on a target tracking algorithm to generate multiple smoke trajectories representing the motion state of the potential smoke area; and a detection module for extracting corresponding continuous image sequences from the video stream based on the multiple smoke trajectories to construct video sequence samples, and processing the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples represent real smoke phenomena.

[0013] This application proposes a video sequence-based method and apparatus for detecting fireworks. First, target detection processing is performed on a single frame image in a video stream to obtain a set of detection boxes containing potential fireworks regions. Then, for each potential fireworks region in the detection box set, multiple consecutive frames of the video stream are processed based on a target tracking algorithm to generate multiple fireworks trajectories representing the motion state of the potential fireworks region. Finally, based on the multiple fireworks trajectories, corresponding continuous image sequences are extracted from the video stream to construct video sequence samples. These video sequence samples are then processed using a pre-trained sequence classification model to determine whether the video sequence sample represents a real fireworks phenomenon. Through the above process, the method and apparatus proposed in this application achieve the technical effect of reducing the false alarm rate and improving the accuracy of recognizing real fireworks dynamic behavior. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0015] Figure 1 is a flowchart illustrating a video sequence-based smoke detection method provided in an embodiment of this application.

[0016] Figure 2 is a schematic diagram of a smoke detection device based on video sequence provided in an embodiment of this application.

[0017] Explanation of reference numerals in the attached figures: Target detection and processing module 10, Target tracking module 20, Detection module 30. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0019] This application provides a smoke detection method based on video sequences, as shown in Figure 1. The method includes:

[0020] Step S100: Perform target detection processing on a single frame image in the video stream to obtain a set of detection boxes containing potential areas of smoke and fire, wherein the video stream originates from a traffic monitoring scene.

[0021] Specifically, a target detection model optimized for fireworks detection is adopted to extract and analyze pixel-level features of the input single-frame image. Through the feature recognition capability of the model, regions that match the features of fireworks, such as open flames and smoke, are selected and the positions of these regions are marked with rectangular detection boxes. At the same time, the category confidence of each detection box is output, that is, the probability that the region is fireworks. Finally, a set of detection boxes containing all potential fireworks regions is formed.

[0022] For example, in highway traffic monitoring videos, if a faint smoke appears next to a vehicle or a small flame appears on the roadside in a single frame image, the model will identify these areas through feature matching and generate corresponding detection boxes. The detection boxes will select the range of smoke or flames, and will also include confidence values ​​such as 0.75 and 0.82 for further judgment.

[0023] In one possible implementation, target detection processing is performed on a single frame image in the video stream to obtain a set of detection boxes containing potential smoke and fire regions. Step S100 further includes step S110, which uses a target detection model based on the YOLO architecture to perform target detection processing on the single frame image in the video stream and outputs the set of detection boxes containing the location of the bounding boxes of potential smoke and fire regions and the category confidence. The backbone network of the target detection model uses a C3K2 module that integrates CSPNet and depthwise separable convolutions for feature extraction, performs multi-scale feature fusion through SPPF layers, uses a PAN structure as the neck network to enhance feature aggregation, introduces the CBAM attention mechanism in the feature fusion stage, uses a decoupled head design to separate classification and regression tasks, and applies an anchor-free mechanism to directly predict bounding boxes.

[0024] Specifically, a YOLO architecture model with a specific structure is used for single-frame image smoke and fire potential region detection. First, the C3K2 module of the backbone network combines the feature splitting and fusion capabilities of the cross-stage local network CSPNet with the lightweight characteristics of depthwise separable convolution to extract features layer by layer from the image, improving feature extraction efficiency while reducing the number of model parameters. Next, the SPPF layer processes the feature map output by the backbone network through pooling operations of different sizes, fusing multi-scale feature information, enabling the model to identify targets of different sizes, from initial small fires to large-area smoke and fire. Then, the PAN structure in the neck enhances the aggregation effect of features at different levels through bidirectional path transmission from top to bottom and bottom to top, focusing on improving the detection accuracy of small target smoke and fire. During feature fusion, the CBAM attention mechanism automatically calculates the importance weights of each channel and spatial position of the feature map, focusing on key feature regions of flames and smoke and suppressing background interference. The detection head adopts a decoupled design, separating the classification task of determining whether it is smoke and fire from the regression task of determining the position and size of the detection box, optimizing each separately to improve detection accuracy. Meanwhile, abandoning the traditional method of pre-setting anchor boxes, the method directly predicts the bounding boxes of the fireworks region based on image feature points through the Anchor-Free mechanism, reducing the impact of hyperparameter adjustment. The final output is a set of detection boxes containing the bounding box coordinates of each potential fireworks region, such as the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, the height, and the class confidence.

[0025] For example, when processing nighttime images from urban road surveillance, the C3K2 module efficiently extracts smoke features from the background of streetlights. The SPPF layer fuses smoke features at different scales, the PAN structure enhances the aggregation of features from weak smoke, the CBAM attention mechanism focuses on the smoke area while ignoring streetlight interference, the decoupling head determines that the area is smoke and determines the bounding box position, and the Anchor-Free mechanism directly generates a bounding box that fits the smoke contour. The output is a detection result with bounding box coordinates (120, 350, 80, 60) and a confidence level of 0.78.

[0026] Step S200: For each potential fire area in the detection box set, process multiple consecutive frames of the video stream based on the target tracking algorithm to generate multiple fire trajectories representing the motion state of the potential fire area.

[0027] Specifically, a multi-target tracking algorithm based on detection results is adopted. The set of detection boxes output in step S100 is used as the initial tracking target. In subsequent consecutive frames of images, the detection boxes of the same potential fire area in each frame are associated, and the changes in their position, size, confidence and other information over time are recorded to form coherent trajectory data. Each trajectory corresponds to the movement process of a potential fire area. At the same time, the algorithm handles situations such as trajectory breakage and the addition of new potential areas to ensure the continuity and integrity of the tracking.

[0028] For example, in a traffic surveillance video, a potential smoke area is located on the left side of the image in the first frame, moves slightly to the upper right and expands in the second frame, and is partially obscured by vehicles in the third frame. The tracking algorithm generates a trajectory that includes information on the location and size changes of the area in these three frames and subsequent frames by associating the features and positional relationships of the area in each frame, thus fully recording its movement and diffusion state.

[0029] In one possible implementation, for each potential firework area in the detection box set, multiple consecutive frames of the video stream are processed based on a target tracking algorithm to generate multiple firework trajectories representing the motion state of the potential firework area. Step S200 further includes step S210, which divides the detection box set in the current frame into a high-confidence detection box set and a low-confidence detection box set based on a confidence threshold. Specifically, two confidence thresholds are preset: a high-confidence threshold (e.g., 0.6) and a low-confidence threshold (e.g., 0.1). Each detection box in the current frame's detection box set is traversed. Detection boxes with a confidence level greater than the high-confidence threshold are assigned to the high-confidence detection box set, and detection boxes with a confidence level between the low-confidence threshold and the high-confidence threshold are assigned to the low-confidence detection box set. Detection boxes with a confidence level lower than the low-confidence threshold are discarded and do not participate in subsequent tracking and matching.

[0030] For example, if a high confidence threshold of 0.6 and a low confidence threshold of 0.1 are set, in the current frame detection box, detection boxes with confidence of 0.72 and 0.85 are assigned to the high confidence set, detection boxes with confidence of 0.23 and 0.51 are assigned to the low confidence set, and detection boxes with confidence of 0.08 are discarded.

[0031] Step S220: Kalman filtering is used to predict the positions of multiple fireworks trajectories from the previous frame in the current frame, resulting in a set of predicted trajectory boxes. Specifically, based on the position, velocity, acceleration, and other state information of each fireworks trajectory in the previous frame, the prediction equation of the Kalman filter is used to estimate the possible position and range of the potential fireworks area corresponding to that trajectory in the current frame, generating predicted trajectory boxes. Each predicted box contains predicted bounding box coordinates, which are used to match the actual detection boxes in the current frame.

[0032] Step S230 involves a two-stage matching process between the trajectory prediction box set, the high-confidence detection box set, and the low-confidence detection box set, outputting multiple fireworks trajectories after processing the current frame. Specifically, in the first stage, the trajectory prediction boxes are matched with the high-confidence detection boxes. The intersection-over-union ratio (IoU) is used to measure the overlap between the prediction boxes and detection boxes, and the optimal matching relationship is found using the Hungarian algorithm. The state of the matched trajectories is then updated. In the second stage, for the trajectories and detection boxes that were not matched in the first stage, the unmatched trajectories are matched again with the low-confidence detection boxes. Firework areas whose confidence may have decreased due to occlusion or changes in lighting are added and associated. Finally, the matched trajectories and the newly added trajectories are integrated to form a complete fireworks trajectory set for the current frame.

[0033] In one possible implementation, a two-stage matching process is performed between the trajectory prediction box set, the high-confidence detection box set, and the low-confidence detection box set to output multiple fireworks trajectories after the current frame processing is completed. Step S230 further includes step S231, calculating the intersection-union ratio (IUGR) of the trajectory prediction box set and the high-confidence detection box set, using the Hungarian algorithm for the first round of matching, associating successfully matched trajectories with their corresponding detection boxes, updating the trajectory status, and obtaining a set of matched trajectories; adding unmatched trajectories and unmatched high-confidence detection boxes to the unmatched trajectory set and the unmatched high-confidence detection box set, respectively. Specifically, the IUGR of each trajectory prediction box and each high-confidence detection box is calculated, where IUGR = area of ​​overlapping region of two boxes / area of ​​joint region of two boxes, resulting in an IUGR matrix. The Hungarian algorithm is used to perform optimal matching calculation on this matrix to find a one-to-one correspondence between trajectory prediction boxes and high-confidence detection boxes. When the IUGR is greater than a set threshold, such as 0.5, it is considered a successful match. For successfully matched trajectories and detection boxes, the position and confidence level of the detection boxes are integrated into the trajectory data to update the current state of the trajectory, including the latest position and motion parameters, forming a set of matched trajectories. Trajectories without matching detection boxes are added to the set of unmatched trajectories, and high-confidence detection boxes without matching trajectories are added to the set of unmatched high-confidence detection boxes.

[0034] For example, the intersection-union ratio (IU) of trajectory prediction box A with high-confidence detection box 1 is 0.62, and the IU of trajectory prediction box B with high-confidence detection box 2 is 0.58, both greater than the threshold of 0.5. Therefore, they are successfully matched using the Hungarian algorithm, and the states of trajectories A and B are updated, forming a set of matched trajectories. Trajectories prediction box C did not find a high-confidence detection box with an IU greater than 0.5 and is added to the set of unmatched trajectories. Similarly, high-confidence detection box 3 did not find a matching trajectory prediction box and is added to the set of unmatched high-confidence detection boxes.

[0035] Step S232: Create a new trajectory for each detection box in the set of unmatched high-confidence detection boxes to obtain a new trajectory set. Specifically, for each detection box in the set of unmatched high-confidence detection boxes, initialize a new trajectory based on its bounding box position, category confidence, and other information in the current frame. Set the initial state parameters of the trajectory, such as initial position, movement speed of 0, trajectory lifetime count, etc. Each new trajectory corresponds to a newly added potential pyrotechnic area in the current frame. All newly initialized trajectories together form a new trajectory set.

[0036] For example, in the set of unmatched high-confidence detection boxes, there are detection boxes 4 (coordinates 180, 220, 50, 45, confidence 0.73) and 5 (coordinates 320, 410, 70, 55, confidence 0.68). Create new trajectories D and E for these two detection boxes respectively. The initial position of trajectory D is (180, 220, 50, 45) and the initial position of trajectory E is (320, 410, 70, 55), forming a new trajectory set.

[0037] Step S233: Calculate the intersection-union ratio (IUR) between the unmatched trajectory set and the low-confidence detection box set. Use the Hungarian algorithm for a second round of matching, associate successfully matched trajectories and their corresponding detection boxes, update the trajectory status, and add them to the matched trajectory set; discard unmatched trajectories and detection boxes. Specifically, calculate the IUR between each trajectory in the unmatched trajectory set and each detection box in the low-confidence detection box set, using an IUR greater than a set threshold as the matching criterion. Then, use the Hungarian algorithm to perform optimal matching between unmatched trajectories and low-confidence detection boxes. For successfully matched trajectories, integrate the corresponding low-confidence detection box information into the trajectory, update the trajectory's current position, motion state, and other parameters, and move the trajectory from the unmatched trajectory set to the matched trajectory set. Unmatched trajectories may indicate that the actual fireworks area has disappeared, and low-confidence detection boxes may be false detections; therefore, unmatched trajectories and low-confidence detection boxes are discarded and no longer participate in subsequent tracking.

[0038] Step S234: All trajectories in the matched trajectory set and the new trajectory set are output as multiple fireworks trajectories after the current frame is processed, for continuous tracking processing of the next frame of the video stream. Specifically, the matched trajectory set updated after successful matching in the first stage is merged with the matched trajectory updated after supplementary matching in the second stage, as well as the newly created trajectory set, to form a set of all valid fireworks trajectories in the current frame. Each trajectory contains complete information such as position, motion state, and confidence changes up to the current frame. This set is output as the basis for trajectory prediction and matching in the next frame image, ensuring continuous tracking of potential fireworks areas.

[0039] Step S300: Based on the multiple fireworks trajectories, extract the corresponding continuous image sequences from the video stream to construct video sequence samples, and process the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples are real fireworks phenomena.

[0040] Specifically, based on the temporal range of each fireworks trajectory (i.e., the video frame interval covered by the trajectory) and its spatial location (i.e., the detection box region corresponding to the trajectory), continuous image frames containing the complete motion process of the trajectory are extracted from the video stream, ensuring consistent time intervals between frames, such as ≤1 / 15 seconds, to construct video sequence samples corresponding to each trajectory. These video sequence samples are then input into a pre-trained sequence classification model. The model analyzes the spatiotemporal features and dynamic changes in the sequence samples to determine whether the trajectory corresponding to the sequence is a real firework, such as an open flame or smoke, and outputs a judgment result of real or non-real fireworks.

[0041] For example, a certain trajectory covers the 10th to 25th frames of the image, and its spatial location is concentrated in the right lane area of ​​the image. Continuous segments of this area are extracted from these 16 frames of the video stream to construct video sequence samples. After being input into the classification model, the model analyzes that there are dynamic features of smoke gradually spreading in the sequence and determines that it is real fireworks.

[0042] In one possible implementation, based on the multiple fireworks trajectories, corresponding continuous image sequences are extracted from the video stream to construct video sequence samples. These video sequence samples are then processed using a pre-trained sequence classification model to determine whether they represent a real fireworks phenomenon. Step S300 further includes step S310, where corresponding continuous image frames are extracted from the video stream based on the temporal range and spatial location covered by each fireworks trajectory to construct video sequence samples. Specifically, the temporal information of each fireworks trajectory is extracted to determine the frame number at which the trajectory begins and ends, and to determine the continuous frame interval covered by the trajectory. Then, the spatial location information of the trajectory in each frame, i.e., the coordinates of the corresponding detection box, is extracted. Next, according to the frame number order, the region image selected by the detection box in each frame is extracted sequentially from the video stream to ensure that the extracted image segments are continuous and the time interval between frames is uniform, avoiding dynamic feature distortion due to inconsistent frame intervals. Finally, the captured continuous region images are combined in frame order to form video sequence samples corresponding to each trajectory. The sample length can be set uniformly, such as 16 frames. If the length is insufficient, it can be processed by frame padding. If the length is excessive, the core frame segment can be extracted.

[0043] Step S320: The video sequence samples are input into a sequence classification model based on a 3D convolutional neural network for processing, and the result is output to determine whether the trajectory corresponding to the video sequence sample is a real fireworks phenomenon. The 3D convolutional neural network uses 3D convolutional kernels to simultaneously extract the spatial and temporal features of the video sequence samples to fuse spatiotemporal information and model the dynamic characteristics of fireworks. The sequence classification model is trained using a video sequence dataset containing positive samples of real fireworks phenomena and negative samples of easily confused interference, with a balanced ratio of positive and negative samples during training.

[0044] Specifically, the core of the 3D convolutional neural network is the use of a three-dimensional convolutional kernel, such as 3×3×3. The first two dimensions correspond to the width and height of the image, and the third dimension corresponds to the temporal dimension of the frame sequence. Convolution operations are performed on the input video sequence samples to extract spatial features of each frame, such as the shape and color of fireworks, while simultaneously capturing temporal features between frames, such as the diffusion, flickering, and movement of fireworks, achieving deep fusion of spatiotemporal features. The model gradually extracts high-level semantic features through multi-layer 3D convolution and pooling operations, modeling the dynamic characteristics of fireworks and distinguishing the dynamic differences between real fireworks and distracting objects. The training dataset for this classification model contains two classes of samples: positive samples are video sequences of real fireworks phenomena, including open flames and smoke sequences with different scenes, fire sizes, and smoke concentrations; negative samples are video sequences of distracting objects that are easily confused with fireworks, such as car headlight reflections, cloud shadow movement, and light flickering. During training, the ratio of positive to negative samples is ensured to be 1:1 to avoid bias in the model towards one class of samples. During training, mixed precision training and distributed training are used to improve training efficiency. The model parameters are adjusted by monitoring the accuracy and F1 score of the validation set. If the F1 score does not improve for 10 consecutive rounds, the training is terminated. The finally trained model can output the judgment result of real or non-real fireworks based on the input video sequence samples.

[0045] For example, if a video sequence sample containing the dynamics of smoke diffusion is input into the model, the 3D convolution kernel will simultaneously extract the spatial shape features and temporal diffusion features of the smoke. After fusion, the features will be matched with the features of real smoke samples in the training set, and the result of judging real smoke will be output. If the input is a video sequence sample of car headlight reflection, the model will extract its spatial color features and temporal flicker features through 3D convolution, and match them with the car headlight interference features in the negative samples, and the result of judging non-real smoke will be output.

[0046] This application employs target detection on single-frame images of a video stream to obtain a set of detection boxes for potential fireworks areas. A target tracking algorithm tracks each potential area across multiple consecutive frames, generating corresponding fireworks trajectories. Based on these trajectories, continuous image sequences are extracted to construct samples. These samples are then processed by a pre-trained sequence classification model to determine whether they represent real fireworks phenomena. These techniques address the technical problems of high false alarm rates and insufficient recognition of dynamic fireworks characteristics in existing fireworks detection methods, achieving the technical effect of reducing false alarm rates and improving the accuracy of recognizing real dynamic fireworks behavior.

[0047] In the preceding text, a smoke detection method based on a video sequence according to an embodiment of the present invention was described in detail with reference to FIG1. ​​Next, a smoke detection device based on a video sequence according to an embodiment of the present invention will be described with reference to FIG2.

[0048] A video sequence-based smoke detection device according to an embodiment of the present invention addresses the technical problems of high false alarm rate and insufficient identification of dynamic smoke characteristics in existing smoke detection methods, achieving the technical effect of reducing the false alarm rate and improving the accuracy of identifying real dynamic smoke behavior. The video sequence-based smoke detection device includes: a target detection and processing module 10, a target tracking module 20, and a detection module 30.

[0049] The target detection processing module 10 is used to perform target detection processing on a single frame image in the video stream to obtain a set of detection boxes containing potential fireworks areas; the target tracking module 20 is used to process multiple consecutive frames of the video stream for each potential fireworks area in the detection box set based on a target tracking algorithm to generate multiple fireworks trajectories representing the motion state of the potential fireworks area; the detection module 30 is used to extract corresponding continuous image sequences from the video stream based on the multiple fireworks trajectories to construct video sequence samples, and process the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples are real fireworks phenomena.

[0050] The specific configuration of the target detection processing module 10 is described in detail below: As mentioned above, target detection processing is performed on a single frame image in the video stream to obtain a set of detection boxes containing potential smoke and fire regions. The target detection processing module 10 may further include: using a target detection model based on the YOLO architecture to perform target detection processing on a single frame image in the video stream, and outputting the set of detection boxes containing the location of the bounding boxes of potential smoke and fire regions and the category confidence; wherein, the backbone network of the target detection model uses a C3K2 module integrating CSPNet and depthwise separable convolution for feature extraction, performs multi-scale feature fusion through SPPF layers, uses a PAN structure as the neck network to enhance feature aggregation, introduces the CBAM attention mechanism in the feature fusion stage, uses a decoupled head design to separate classification and regression tasks, and applies an Anchor-Free mechanism to directly predict bounding boxes.

[0051] The target tracking module 20 is described in detail below: As mentioned above, for each potential firework area in the detection box set, the video stream is processed using a target tracking algorithm across multiple consecutive frames to generate multiple firework trajectories representing the motion state of the potential firework area. The target tracking module 20 may further include: a confidence division unit for dividing the detection box set in the current frame into a high-confidence detection box set and a low-confidence detection box set based on a confidence threshold; a trajectory prediction unit for using Kalman filtering to predict the positions of multiple firework trajectories from the previous frame in the current frame, thus obtaining a trajectory prediction box set; and a two-stage matching unit for performing a two-stage matching between the trajectory prediction box set and the high-confidence detection box set and the low-confidence detection box set, outputting multiple firework trajectories after the current frame has been processed.

[0052] The process involves a two-stage matching of the trajectory prediction box set, the high-confidence detection box set, and the low-confidence detection box set to output multiple fireworks trajectories after the current frame is processed. The two-stage matching unit may further include: a first-round matching subunit for calculating the intersection-union ratio (IUU) of the trajectory prediction box set and the high-confidence detection box set, performing the first round of matching using the Hungarian algorithm, associating successfully matched trajectories with their corresponding detection boxes, updating the trajectory status, and obtaining a set of matched trajectories; and adding unmatched trajectories and unmatched high-confidence detection boxes to the unmatched trajectory set and the unmatched high-confidence detection box set, respectively. The set of detection boxes is used to create a new trajectory for each detection box in the set of unmatched high-confidence detection boxes, resulting in a new trajectory set. The second-round matching subunit is used to calculate the intersection-union ratio (IUGR) between the set of unmatched trajectories and the set of low-confidence detection boxes, and to perform a second round of matching using the Hungarian algorithm. The successfully matched trajectories and their corresponding detection boxes are associated, the trajectory status is updated, and they are added to the set of matched trajectories. Unmatched trajectories and detection boxes are discarded. All trajectories in the set of matched trajectories and the new trajectory set are output as multiple fireworks trajectories after the current frame is processed, for continuous tracking processing of the next frame of the video stream.

[0053] The specific configuration of the detection module 30 is described in detail below: As mentioned above, based on the multiple fireworks trajectories, corresponding continuous image sequences are extracted from the video stream to construct video sequence samples, and the video sequence samples are processed using a pre-trained sequence classification model to obtain a determination result of whether the video sequence samples are real fireworks phenomena. The detection module 30 may further include: a video sequence sample construction unit used to extract corresponding continuous image frames from the video stream according to the temporal range and spatial location covered by each fireworks trajectory to construct video sequence samples; a sequence classification unit used to input the video sequence samples into a sequence classification model based on a 3D convolutional neural network for processing, and output a determination result of whether the trajectory corresponding to the video sequence sample is a real fireworks phenomenon; wherein, the 3D convolutional neural network uses 3D convolutional kernels to simultaneously extract the spatial and temporal dimension features of the video sequence samples to fuse spatiotemporal information and model the dynamic characteristics of fireworks.

[0054] The sequence classification unit may further include: the sequence classification model is trained using a video sequence dataset containing positive samples of real fireworks phenomena and negative samples of easily confused interference, and the ratio of positive to negative samples is balanced during training.

[0055] The target detection and processing module 10 may further include: the video stream originating from a traffic monitoring scenario.

[0056] The smoke and fire detection device based on video sequence provided in this embodiment of the invention can execute the smoke and fire detection method based on video sequence provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0057] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for detecting smoke and fire based on video sequences, characterized in that, The method includes: performing target detection processing on a single frame image in a video stream to obtain a set of detection boxes containing potential fireworks regions; for each potential fireworks region in the detection box set, processing multiple consecutive frames of the video stream based on a target tracking algorithm to generate multiple fireworks trajectories representing the motion state of the potential fireworks region; based on the multiple fireworks trajectories, extracting corresponding continuous image sequences from the video stream to construct video sequence samples, and processing the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples represent real fireworks phenomena.

2. The smoke and fire detection method based on video sequences as described in claim 1, characterized in that, Object detection processing is performed on single-frame images in a video stream to obtain a set of detection boxes containing potential fireworks regions. This includes: using a YOLO-based object detection model to perform object detection processing on single-frame images in the video stream, and outputting the set of detection boxes containing the location of the bounding boxes of potential fireworks regions and their category confidence scores; wherein, the backbone network of the object detection model uses a C3K2 module integrating CSPNet and depthwise separable convolutions for feature extraction, performs multi-scale feature fusion through SPPF layers, uses a PAN structure as the neck network to enhance feature aggregation, introduces the CBAM attention mechanism in the feature fusion stage, uses a decoupled head design to separate classification and regression tasks, and applies an anchor-free mechanism to directly predict bounding boxes.

3. The smoke and fire detection method based on video sequences as described in claim 1, characterized in that, For each potential firework region in the detection box set, the video stream is processed using a target tracking algorithm across multiple consecutive frames to generate multiple firework trajectories representing the motion state of the potential firework region. This process includes: dividing the detection box set in the current frame into a high-confidence detection box set and a low-confidence detection box set based on a confidence threshold; using Kalman filtering to predict the positions of multiple firework trajectories from the previous frame in the current frame to obtain a trajectory prediction box set; and performing a two-stage matching between the trajectory prediction box set and the high-confidence and low-confidence detection box sets to output the multiple firework trajectories after processing in the current frame.

4. The smoke and fire detection method based on video sequences as described in claim 3, characterized in that, The trajectory prediction box set, the high-confidence detection box set, and the low-confidence detection box set are matched in a two-stage process to output multiple fireworks trajectories after the current frame is processed. This includes: calculating the intersection-union ratio (IUU) of the trajectory prediction box set and the high-confidence detection box set; performing a first round of matching using the Hungarian algorithm; associating successfully matched trajectories with their corresponding detection boxes; updating the trajectory status to obtain a set of matched trajectories; adding unmatched trajectories and unmatched high-confidence detection boxes to the unmatched trajectory set and the unmatched high-confidence detection box set, respectively; and then... For each unmatched high-confidence detection box in the set of unmatched detection boxes, a new trajectory is created, resulting in a new trajectory set. The intersection-union ratio (IUGR) of the unmatched trajectory set and the low-confidence detection box set is calculated, and a second round of matching is performed using the Hungarian algorithm. The successfully matched trajectories and their corresponding detection boxes are associated, the trajectory status is updated, and they are added to the set of matched trajectories. Unmatched trajectories and detection boxes are discarded. All trajectories in the set of matched trajectories and the new trajectory set are output as multiple fireworks trajectories after the current frame is processed, for continuous tracking processing of the next frame of the video stream.

5. The smoke and fire detection method based on video sequences as described in claim 1, characterized in that, Based on the multiple fireworks trajectories, corresponding continuous image sequences are extracted from the video stream to construct video sequence samples. These video sequence samples are then processed using a pre-trained sequence classification model to determine whether they represent real fireworks phenomena. This process includes: extracting corresponding continuous image frames from the video stream based on the temporal range and spatial location covered by each fireworks trajectory to construct video sequence samples; inputting the video sequence samples into a sequence classification model based on a 3D convolutional neural network for processing, and outputting a determination result indicating whether the corresponding trajectory of the video sequence sample represents a real fireworks phenomenon; wherein the 3D convolutional neural network uses 3D convolutional kernels to simultaneously extract the spatial and temporal dimensions of the video sequence samples to fuse spatiotemporal information and model the dynamic characteristics of fireworks.

6. The smoke and fire detection method based on video sequences as described in claim 5, characterized in that, The sequence classification model is trained using a video sequence dataset containing positive samples of real fireworks phenomena and negative samples of easily confused interference, and the ratio of positive to negative samples is balanced during training.

7. The smoke and fire detection method based on video sequences as described in claim 1, characterized in that, The video stream originates from a traffic monitoring scenario.

8. A smoke and fire detection device based on video sequences, characterized in that, The apparatus is used to implement a video sequence-based smoke detection method according to any one of claims 1-7. The apparatus includes: a target detection processing module, used to perform target detection processing on a single frame image in a video stream to obtain a set of detection boxes containing potential smoke areas; a target tracking module, used to process multiple consecutive frames of the video stream for each potential smoke area in the set of detection boxes based on a target tracking algorithm to generate multiple smoke trajectories representing the motion state of the potential smoke area; and a detection module, used to extract corresponding continuous image sequences from the video stream based on the multiple smoke trajectories to construct video sequence samples, and to process the video sequence samples using a pre-trained sequence classification model to obtain a determination result on whether the video sequence samples are real smoke phenomena.

Images