A video smoke detection method based on time-series dynamic features and related devices

Abstract

This application provides a video smoke detection method and related equipment based on temporal dynamic features, belonging to the fields of computer vision and intelligent transportation technology. The method includes: performing target detection on image frames in an input video stream to obtain suspected smoke regions; performing target tracking on the suspected smoke regions in multiple consecutive image frames, generating a unique spatiotemporal trajectory for each suspected smoke region; extracting image features corresponding to each image frame in the spatiotemporal trajectory to construct a feature sequence; inputting the feature sequence into a pre-trained temporal classification model to output the smoke category or false detection category corresponding to the spatiotemporal trajectory. This application generates spatiotemporal trajectories by fusing target detection and multi-target tracking, extracts multi-source features that fuse visual and frequency domain features, and uses a Transformer-based self-attention mechanism for feature importance filtering, ultimately achieving accurate differentiation between smoke and interference objects through a temporal classification model.

Description

Technical Field

[0001] This application relates to the fields of computer vision and intelligent transportation technology, and in particular to a video smoke detection method and related equipment based on temporal dynamic features. Background Technology

[0002] With the rapid expansion of the expressway network, its importance as the artery of the national economy is becoming increasingly prominent. However, in the closed, high-speed environment, sudden incidents such as fires can easily lead to serious consequences, posing a severe challenge to public safety. Therefore, utilizing intelligent methods for early warning of expressway risks has become an urgent need for industry development. Among the many risks, vehicle fires are particularly dangerous due to their rapid onset and spread. Because of obstructed monitoring angles or vehicle obstructions, the initial open flames of a fire are often difficult to detect directly, making reliable smoke detection the most critical and effective technical approach for achieving early fire warnings.

[0003] In recent years, object detection algorithms based on convolutional neural networks (CNNs), especially one-stage detectors represented by the YOLO series, have been widely used in various visual tasks due to their good balance between accuracy and speed. In the field of smoke detection, models such as YOLO, trained on a large amount of labeled data, can locate and classify smoke regions in a single frame image, initially realizing the leap from manual monitoring to automatic alarm, and greatly improving monitoring efficiency.

[0004] However, deploying such general object detection models based on single-frame images directly in the complex and ever-changing real-world scenarios of highways reveals their limitations, namely, the potential for false alarms in complex environments. Highway monitoring environments are extremely complex, with factors such as day-night cycles, weather changes (rain, snow, fog), lighting conditions (strong light, backlight, nighttime headlights), and the condition of the cameras themselves (stains, water droplets) causing numerous interfering elements in the images that resemble the visual characteristics of smoke. Examples include headlight spots reflected from wet, slippery roads in rainy weather, diffuse halos from distant headlights at night, and water droplets or stains adhering to the lens. In a single static image, the color, texture, and blurriness are highly similar to early smoke, making detection models relying solely on spatial appearance features prone to misjudgment. Current mainstream single-frame detection methods sever the connection between frames, failing to utilize the crucial criterion of "smoke's unique diffusion and movement patterns over time," thus leading to false detections at their source.

[0005] To address the false alarm problem in single-frame detection, academia and industry have explored various approaches. Traditional methods, such as inter-frame differencing and optical flow, attempt to incorporate motion information. However, in highway scenarios, cameras may experience global motion due to high-speed vehicle traffic or camera shake. Furthermore, smoke diffusion is slow and non-rigid, making traditional optical flow algorithms sensitive to noise, computationally complex, and unable to accurately capture and characterize the subtle dynamics of smoke, thus limiting their practicality. Another approach is to construct a larger dataset covering more interference scenarios to improve model robustness, but this incurs high data annotation costs. Summary of the Invention

[0006] The purpose of this application is to overcome the high false alarm rate of existing smoke detection methods based on single-frame images in complex real-world highway scenarios. It proposes a video smoke detection method, system, electronic device, storage medium, and program product based on temporal dynamic features. This application aims to model the unique dynamic behavior characteristics of smoke over time, such as diffusion and irregular movement, by integrating single-frame detection and temporal analysis. This effectively distinguishes real smoke from highly similar static or regularly moving interference such as road surface reflections, headlight halos, and lens smudges. This application strives to significantly reduce the false alarm rate of the system while ensuring real-time performance, thereby improving the reliability and practicality of the smoke warning system.

[0007] To achieve the above objectives, one aspect of this application proposes a video smoke detection method based on temporal dynamic features, the method comprising: Perform target detection on image frames in the input video stream to obtain suspected smoke regions in each frame; Target tracking is performed on the suspected smoke regions in multiple consecutive frames of images, and a unique spatiotemporal trajectory is generated for each suspected smoke region; For each of the aforementioned spatiotemporal trajectories, image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory are extracted, and a feature sequence is constructed based on the image features; The feature sequence is input into a pre-trained temporal classification model, and the temporal classification model outputs the category corresponding to the spatiotemporal trajectory, which includes smoke category or false detection category.

[0008] In some embodiments, extracting image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory, and constructing a feature sequence based on the image features, includes: For each frame in the spatiotemporal trajectory, multiple features are extracted from the image region corresponding to the suspected smoke area, and the multiple features are concatenated to form a single-frame original feature vector. The original feature vector of the single frame is standardized to obtain the standardized single frame feature vector; The standardized single-frame feature vectors of L consecutive frames in the spatiotemporal trajectory are arranged in chronological order to construct the feature sequence, where L is the preset time window length.

[0009] In some embodiments, the multiple features include visual features and frequency domain features; The visual features include: hierarchical features extracted from the feature pyramid of the target detection model, and channel importance features determined based on the average activation intensity of each channel in the feature pyramid within the suspected smoke area; The frequency domain features include: frequency domain statistical features extracted by performing Fast Fourier Transform on the image region, frequency domain structural features extracted by performing Discrete Cosine Transform, and texture roughness features extracted based on frequency domain gradient.

[0010] In some embodiments, a feature importance screening step is also included: A time-series feature importance analysis model based on Transformer self-attention mechanism is used to process a training dataset consisting of multiple feature sequences, learn the contribution of each feature dimension in the time-series evolution, and select a preset number of discriminative feature dimensions from all feature dimensions based on the contribution. The construction of the feature sequence specifically involves: extracting the corresponding feature values ​​from the standardized single-frame feature vector based on the selected discriminative feature dimensions, and constructing a simplified feature sequence.

[0011] In some embodiments, the processing procedure of the temporal feature importance analysis model based on the Transformer self-attention mechanism includes: The input feature sequence is passed through a multi-head self-attention layer, and the multi-head attention score matrix is ​​calculated. The multi-head attention score matrix is ​​averaged element-wise to obtain a global attention matrix that integrates multi-head information. The global attention matrix is ​​averaged over the time dimension to obtain the temporal importance score for each feature dimension; The feature dimensions are sorted according to the temporal importance score, and a preset number of feature dimensions with the highest ranking are selected as the discriminative feature dimensions.

[0012] In some embodiments, the time-series classification model is a classification model built on an Encoder-Only Transformer architecture; The step of inputting the feature sequence into a pre-trained temporal classification model and outputting the category corresponding to the spatiotemporal trajectory through the temporal classification model includes: A learnable classification label is added before the feature sequence and then added to the positional encoding before being input into a multi-layer Transformer encoder. Obtain the enhanced feature sequence output by the multi-layer Transformer encoder, the enhanced feature sequence including the enhanced classification label vector and the enhanced feature vectors at each time step; The enhanced classification label vector is fused with the mean vector obtained by global average pooling of the enhanced feature vectors at each time step to obtain the fused feature vector. The fused feature vector is input into the classification head to obtain the probability that the spatiotemporal trajectory belongs to the smoke category or the false detection category.

[0013] To achieve the above objectives, another aspect of this application proposes a video smoke detection system based on temporal dynamic features, comprising: The target detection and trajectory generation module is used to perform target detection on image frames in the input video stream, obtain the suspected smoke region in each frame, and perform target tracking on the suspected smoke region in multiple consecutive frames, generating a unique spatiotemporal trajectory for each suspected smoke region. The feature extraction and processing module is used to extract the image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory for each spatiotemporal trajectory, and construct a feature sequence based on the image features; The temporal modeling and decision module is used to input the feature sequence into a pre-trained temporal classification model and output the category corresponding to the spatiotemporal trajectory through the temporal classification model. The category includes smoke category or false detection category.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0016] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0017] Compared with the prior art, this application has the following advantages and beneficial effects: 1) Significantly reduce false alarm rate: This application introduces a three-level framework of "detection-tracking-temporal analysis" to deeply mine and utilize the unique dynamic behavioral characteristics of smoke in the time dimension, such as diffusion and irregular movement. It can effectively distinguish real smoke from highly similar static or regularly moving interference objects such as car headlight halos at night, road surface reflections on rainy days, and water droplets on lenses. This significantly filters false alarms caused by false detections at the source, solving the core pain point of high false alarm rate in actual deployment of existing technologies.

[0018] 2) Strong feature representation capability: This application designs a multi-source feature fusion strategy, which not only extracts visual semantic features from the feature pyramid of the target detection model, but also introduces frequency domain features (FFT, DCT, frequency domain gradient) to characterize the target characteristics from the transform domain, and constructs rich original feature vectors, providing a more comprehensive information foundation for subsequent time series analysis.

[0019] 3) Automated feature selection: This application innovatively proposes a sequence-level feature importance analysis and selection mechanism based on Transformer self-attention, which can automatically learn the contribution of each feature dimension to time series discrimination from end to end, realize intelligent dimensionality reduction and optimization of feature dimensions, improve model performance and reduce computational complexity.

[0020] 4) High accuracy in temporal modeling: This application adopts the Encoder-Only Transformer architecture, which captures long-range temporal dependencies in parallel through a self-attention mechanism, and introduces a dual-path feature fusion strategy (global [CLS] label + local GAP) to fully integrate global contextual semantics and local statistical information, thereby further improving the accuracy and robustness of classification.

[0021] 5) Low data annotation cost: The automated training dataset construction method provided in this application can automatically generate temporal feature sequence samples with only coarse annotation at the video level, which greatly reduces the data annotation cost and enables the model to iterate quickly and adapt to massive real-world scenarios. Attached Figure Description

[0022] Figure 1 This is an overall flowchart of the technical solution provided in the embodiments of this application.

[0023] Figure 2 This is a schematic diagram of target detection and trajectory generation tracking in an embodiment of this application, where (a) is the detection result using only the target detection algorithm, and (b) is the result using both the target detection and tracking algorithms.

[0024] Figure 3 This is a flowchart illustrating the extraction of multiple features in an embodiment of this application.

[0025] Figure 4This is a diagram of the temporal feature importance filtering architecture based on Transformer self-attention in the embodiments of this application.

[0026] Figure 5 This is a diagram of the temporal modeling and decision network structure based on Encoder-Only Transformer in the embodiments of this application.

[0027] Figure 6 This is a structural diagram of the timing coding module in an embodiment of this application.

[0028] Figure 7 This is a structural diagram of the timing pooling and classification module in the embodiments of this application.

[0029] Figure 8 This is a comparison diagram of the effects of timing system detection and single-frame detection in the embodiments of this application, where (a) is the single-frame detection result and (b) is the timing system detection result.

[0030] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0031] Figure 10 This is a flowchart of the steps of a video smoke detection method based on temporal dynamic features provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] In summary, current mainstream smoke detection methods based on single-frame images face the challenge of excessively high false alarm rates in complex highway monitoring scenarios. To address this issue, it is necessary to propose an optimization scheme that effectively filters out most false alarms caused by environmental interference without significantly increasing computational burden, making smoke detection more reliable and practical, thus possessing clear research value and engineering application prospects.

[0035] In view of this, this application provides a video smoke detection method, system, electronic device, storage medium, and program product based on temporal dynamic features. Firstly, addressing the problem of high false alarms in complex scenes (such as nighttime or rainy days) caused by the similarity of static features between smoke and interfering objects in existing single-frame detection models based on YOLO, this application proposes a three-level processing framework of "detection-tracking-temporal analysis," aiming to fully utilize the unique dynamic behavior pattern of smoke in the temporal dimension to filter false alarms. The overall workflow is as follows: Figure 1 As shown in the diagram. Specifically, the system first uses an improved YOLO target detector to efficiently screen suspected smoke regions in each frame of the image, quickly locating candidate targets. Next, a multi-target tracking algorithm is used to establish stable cross-frame trajectories for each candidate target, ensuring consistency of spatiotemporal information. Then, the core innovation lies in the system extracting and fusing multi-level spatial and temporal features from the video sequence along each trajectory to construct a feature sequence describing the dynamic evolution of the target. Finally, a Transformer-based temporal encoder is used to perform deep modeling and classification of the feature sequence to determine whether the target on the trajectory is real smoke, thus effectively eliminating false alarms caused by environmental interference while retaining genuine smoke alarms. The entire solution achieves end-to-end processing from raw video to final judgment in engineering, and designs a process that can automatically construct a temporal feature dataset from video, significantly reducing the data annotation cost for model optimization and iteration.

[0036] The technical solution of this application is mainly achieved by the sequential collaboration of the following core modules: 1) Target detection and trajectory generation module This module is responsible for real-time processing of the input video stream, initially locating suspected smoke areas, and assigning and maintaining a unique ID trajectory for each area. Its core is the combination of the YOLO detector and the SORT tracking algorithm to ensure that subsequent temporal feature extraction can be performed on the same target both spatially and temporally.

[0037] 2) Feature extraction and processing module For each target trajectory maintained by the tracking module, at each time step (video frame), a set of feature vectors is extracted from the corresponding image region. These features not only include multi-level visual features obtained from the YOLO backbone network feature pyramid, but also frequency domain features. Through standardization and sequence-level feature importance filtering based on Transformer self-attention, an effective feature representation for temporal modeling is finally formed.

[0038] 3) Time Series Modeling and Decision Module This module receives feature sequences within a time window from each target trajectory. It employs an Encoder-Only Transformer-based model, using a self-attention mechanism to mine the dependencies between step sizes within the sequence to capture dynamic patterns such as the irregular diffusion and ascent of smoke. Finally, a classifier outputs the probability of whether the trajectory represents real smoke, achieving end-to-end temporal recognition and misclassification elimination.

[0039] The video smoke detection method provided in this application relates to the fields of computer vision and intelligent transportation technology. The video smoke detection method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the video smoke detection method, but is not limited to the above forms.

[0040] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0041] like Figure 10 As shown, this embodiment provides a video smoke detection method based on temporal dynamic features, specifically including the following steps: Step S1: Perform target detection on the image frames in the input video stream to obtain the suspected smoke region in each frame; Step S2: Perform target tracking on the suspected smoke regions in multiple consecutive frames of images, and generate a unique spatiotemporal trajectory for each suspected smoke region; Step S3: For each of the spatiotemporal trajectories, extract the image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory, and construct a feature sequence based on the image features; Step S4: Input the feature sequence into a pre-trained temporal classification model, and output the category corresponding to the spatiotemporal trajectory through the temporal classification model. The category includes smoke category or false detection category.

[0042] The solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific application examples.

[0043] (1) Target detection and trajectory generation module This module employs a tracking-by-detection strategy and serves as the entry point and foundation of the entire temporal analysis system. Its task is to locate suspected smoke regions in each frame of the high-speed video stream and assign a unique ID to each region to ensure spatial and temporal consistency in subsequent temporal feature extraction. As the system's preprocessing front-end, its core function is to generate stable and consistent target trajectory sequences from the input video stream for subsequent temporal analysis modules. Specifically, this module utilizes existing efficient target detection models and multi-target tracking algorithms to convert "video frames" into "spatial-temporal trajectories of identified suspected smoke targets."

[0044] For each frame of the input video, a pre-trained object detection model (such as YOLOv8) is used for forward inference, outputting the set of bounding box coordinates for all suspected smoke regions in the current frame, denoted as . ,in The sequence number of the current frame. This indicates the number of suspected smoke regions (smoke target instances) detected in the current frame.

[0045] To distinguish smoke targets at different locations in a video and maintain their temporal continuity, this module introduces multi-target tracking technology. This invention employs the basic framework of the SORT (Simple Online and Realtime Tracking) algorithm, with data association at its core. This module maintains a spatiotemporal trajectory for each target and utilizes the prediction and association mechanism of the SORT algorithm to stably and efficiently match the detection boxes output by the detector (YOLO) in each frame with existing trajectories. Successfully matched detection boxes are used to update the state of the corresponding trajectory, while unmatched new detections are initialized as new trajectories. This process ensures that subsequent temporal feature extraction always targets the same physical target, providing a stable data foundation for temporal analysis.

[0046] After the above processing, the module outputs a spatiotemporal trajectory for each continuously tracked suspected smoke target. Each trajectory... At any time Each is associated with a unique identifier. A bounding box and the original image region in which it is located. This trajectory data is the direct source for subsequent feature extraction and temporal modeling modules.

[0047] This module is designed to serve the subsequent core modules. Firstly, it provides spatial anchors for feature extraction: the bounding boxes of suspected smoke targets detected by the object detection algorithm define the specific image region to be analyzed by the subsequent "feature extraction and processing module," avoiding inefficient computation across the entire image. Next, it ensures the consistency of temporal features: the trajectory tracked by the SORT algorithm ensures consistency from the trajectory... Image regions at different time points ( The features extracted in (,...) all originate from the same physical target, thus giving the subsequently extracted "temporal feature sequence" physical meaning and effectively characterizing the dynamic evolution behavior of that specific target. The visualization results of the target detection and tracking trajectory generation are as follows: Figure 2 As shown.

[0048] (2) Feature extraction and processing module This module is responsible for outputting each target trajectory from the "Target Detection and Trajectory Generation Module". At each time step (frame) of its lifecycle. Above, from its corresponding image region In this process, a set of 256-dimensional feature vectors that can comprehensively characterize the target's appearance, structure, and temporal evolution potential are extracted and constructed, and these vectors are appended to the feature history sequence of the trajectory. When the temporal modeling module needs to analyze the trajectory... At any moment When judging the state, it will extract the sequence of events ending at time t and of length from the feature history sequence of the trajectory. The most recent feature subsequence forms a shape of Fixed-length feature sequence matrix This serves as the input to the final time-series decision-making module. In this way, the system can make real-time judgments based on the target's most recent dynamic behavior. The processing flow of this module includes processing data from image regions... This involves extracting multiple types of features, grouping and standardizing single-step features, and performing importance analysis and screening of sequence-level features.

[0049] 2.1) Multi-class feature extraction For the target i and its bounding box tracked at time t This step extracts multiple types of features from the corresponding image regions and concatenates them into an original feature vector. The process for extracting multiple features is as follows: Figure 3 As shown.

[0050] 2.1.1) Source of main features: Hierarchical features based on the YOLOv8 feature pyramid To efficiently utilize existing computing resources and obtain rich hierarchical semantic information, this application directly extracts features from the feature pyramid generated during the forward inference process of the YOLOv8 model. The YOLOv8 feature pyramid typically contains multiple layers of feature maps, such as P3, P4, and P5. The P3 layer has the highest resolution, which is beneficial for capturing detailed textures; the P4 layer balances resolution and semantics; while the P5 layer has strong semantics but low resolution. This application primarily utilizes the P3 and P4 layers to ensure effective representation of the subtle features of the smoke target.

[0051] Feature maps of layers P3 and P4 and Based on the detection frame respectively Perform the RoI Align operation. This operation pools irregularly sized regions of interest into a fixed-size mesh (e.g., For feature maps (C represents the number of channels), RoI Align detects each channel within the bounding box. Bilinear interpolation sampling is performed in the subgrid to obtain features. After flattening, a 4C-dimensional vector is obtained from both layers P3 and P4. Since the number of channels C in layer P4 is twice that of layer P3, to reduce dimensionality, this embodiment selects the 48 feature values ​​with the strongest spatial response from the features of layer P3 and the 96-dimensional feature values ​​with the strongest response from the features of layer P4. Therefore, a total of [number missing] vectors are obtained from this part. Dimensional features.

[0052] Channel importance features: Semantic channels sensitive to smoke regions and those easily misdetected as smoke regions are likely to be different. To focus on the semantic channels most sensitive to the current target, the average activation intensity of each channel within the detection box is calculated as its importance score. For feature maps... The Each channel, its importance score The calculation is as follows:

[0053] in, For feature map spatial coordinates, bounding box Mapping to feature map The area above, This represents the area of ​​the region. All channels in layers P3 and P4 are scored according to their importance. Sort in descending order and select the channel numbers (i.e., channel indices) of the top 24 channels in layer P3 and the top 48 channels in layer P4 as features. This part contributes... Dimensional features.

[0054] 2.1.2) Frequency Domain Characteristics To comprehensively describe the target characteristics in the transform domain, this embodiment focuses on the detection box region. A series of frequency domain transformations are performed on a grayscale image, and 40-dimensional features are extracted from it. Let the grayscale image be... .

[0055] a) Fast Fourier Transform (FFT) and Fundamental Spectral Features For images After normalization, a Fast Fourier Transform (FFT) is performed to obtain the spectrum. The formula is as follows:

[0056] in Image size, These are frequency domain coordinates. After centering the spectrum (fftshift), it is separated into amplitude spectra. and phase spectrum Directly to the amplitude spectrum Logarithmic transformations are susceptible to extreme values. This embodiment employs a robust processing procedure: first, the amplitude spectrum is cropped based on percentiles (e.g., 1% and 99.9%) to limit outliers; then, the cropped amplitude spectrum is processed... implement The operation was performed to enhance the low-intensity component. .from Extract the mean Standard deviation and total energy From the phase spectrum Extract its mean and standard deviation These characteristics constitute the basic statistical description in the frequency domain.

[0057] b) Structural features based on Discrete Cosine Transform (DCT) To capture the main structural information of image patches more effectively, this embodiment further refines the grayscale image. Perform a two-dimensional discrete cosine transform (DCT). The DCT transform concentrates the energy of the image onto a few low-frequency coefficients; its transformation formula is:

[0058] in, These are the normalization coefficients. After transformation, the coefficients in the upper left corner correspond to the low-frequency components, representing the overall contour and smoothness of the image; extending to the lower right corner, they correspond to the mid-to-high-frequency components, representing the details and edges of the image.

[0059] This embodiment employs a layered region energy extraction strategy after DCT transformation: 1. Low-frequency structural features: A 4×4 region (16 coefficients in total) in the upper left corner of the DCT coefficient matrix was selected, and the absolute values ​​of the coefficients within this region were calculated. This set of features reflects the most dominant smooth structure and overall brightness distribution pattern in the image patch.

[0060] 2. Mid-to-high frequency texture features: To further capture details, a subsequent 4×4 region (e.g., the 4:8, 4:8 region in the coefficient matrix) is selected from the DCT coefficient matrix, avoiding the core low-frequency region in the upper left corner. The absolute values ​​of the coefficients within this region are also calculated. This set of features reflects more subtle texture variations and edge information in the image.

[0061] By combining low-frequency and mid-to-high-frequency DCT features, this invention can simultaneously characterize the overall shape and local detail texture of the smoke area in this part.

[0062] c) Texture roughness features based on frequency domain gradient Amplitude spectrum image The spatial distribution pattern itself also contains important information. To quantify the texture roughness and drastic changes of the amplitude spectrum, this invention will... Treating it as an image, a 3×3 Sobel operator convolution kernel is used to compute... In the horizontal direction ( ) and vertical direction ( The gradient approximation of is calculated using the following formula:

[0063] Next, gradient magnitude and feature extraction are performed: the gradient magnitude of each pixel is calculated. From the gradient magnitude plot The following statistics are extracted as features: 1. Average gradient strength : Reflects the average intensity of change in the overall amplitude spectrum image. The larger the value, the coarser the texture and the more drastic the change.

[0064] 2. Standard deviation of gradient intensity It reflects the spatial dispersion of gradient intensity and can be used to distinguish between uniform changes and patterns with local drastic changes.

[0065] 3. Gradient Energy The sum of squares of the gradient magnitudes represents the total intensity of the change.

[0066] 4. High gradient pixel ratio The gradient magnitude exceeds the average gradient strength. The proportion of pixels reflects the density of significant edges or areas of change.

[0067] In summary, the single-step original feature vector The structure is: 216 (YOLO features) + 40 (frequency domain) = 256 dimensions.

[0068] 2.2) Single-step feature standardization because Features from different sources have different dimensions and numerical ranges, and direct concatenation can lead to difficulties in model training. This invention employs a grouping normalization strategy to divide the 256-dimensional features into G groups according to their source (this embodiment uses the aforementioned P3-layer RoI Align feature group, P3-layer channel importance feature group, P4-layer RoI Align feature group, etc.). For the g-th group of features... Its standardized features The calculation is as follows:

[0069] in, and These are the mean and standard deviation of the features calculated on the training set, respectively. This is to prevent division by zero of small constants (such as...) After standardization, the resulting vector is... .

[0070] 2.3) Sequence-level feature importance analysis and screening To address the issues of feature dimensional redundancy and information loss from manual statistics in smoke time series, this embodiment proposes a sequence-level feature importance analysis and selection method based on the Transformer self-attention mechanism. The architecture diagram of the method is shown below. Figure 4 As shown in the figure. This method abandons the traditional paradigm of manually constructing time-series statistics and ranking features in random forests. Through end-to-end differentiable attention weight learning, it automatically mines the discriminative contribution of each feature dimension in the temporal evolution of smoke, achieving efficient and robust feature dimension selection, and providing highly discriminative and concise feature inputs for subsequent temporal modeling and decision-making modules.

[0071] 2.3.1) Problem Definition and Design Motivation For a continuously tracked target trajectory, its feature extraction module generates a 256-dimensional feature vector for each time step. Time-series modeling requires understanding the trajectory at a given time step. When judging the state, the system will extract a value with that time as the endpoint and a length of [value missing]. The most recent feature sequence. Therefore, given the input time-series feature sequence. Where L is the time window length and C is the feature dimension of a single frame (in this paper) Traditional sequence-level feature importance analysis methods typically involve manually constructing temporal statistical features such as mean, standard deviation, skewness, and difference statistics. The sequence is then compressed into a fixed-length vector, and feature importance is calculated and ranked using traditional machine learning models like random forests. These methods inherently have limitations. Manually constructed statistics can only capture a limited set of temporal patterns, failing to fully preserve the fine-grained temporal dependencies of smoke's dynamic evolution, and are prone to losing crucial discriminative information. Furthermore, the design of these manually constructed statistics relies on domain priors, resulting in insufficient generalization ability in complex scenarios (such as the coexistence of smoke and background interference like clouds and fog). Moreover, the importance scores calculated by random forests are based on static statistical features, completely separate from the gradient flow of subsequent temporal modeling tasks, making end-to-end optimization impossible. To address these issues, this paper leverages the inherent temporal modeling capabilities of the Transformer's self-attention mechanism, embedding feature importance learning into the temporal encoding process. Multi-head self-attention weights directly measure the contribution of each feature dimension to temporal dependencies, achieving task-aligned end-to-end feature importance evaluation, ultimately selecting the most valuable feature subset for misdetection of smoke.

[0072] 2.3.2) The principle of temporal feature importance based on multi-head self-attention The core of Transformer's multi-head self-attention mechanism is to model the dependencies between sequence elements in different subspaces through multiple parallel attention heads. For a given input sequence... (in The hidden layer dimension of the model is used in this paper, which is related to the input feature dimension. (Consistent), firstly, a query is generated through a linear transformation. ,key ,value :

[0073] in The weight matrix is ​​a learnable matrix. These are the subspace dimensions for query keys and values, respectively.

[0074] The output of single-head self-attention is calculated using scaled dot-product attention:

[0075] in Attention score matrix , Indicates the first The time step for the first The dependence strength at each time step; divided by This is to avoid gradient vanishing and ensure training stability. Multi-head self-attention divides the input into h parallel attention heads, each of which independently computes its self-attention and then concatenates the output.

[0076] in , To output a linear transformation matrix, the final output dimension is the same as the input, i.e. .

[0077] Intuitively, if a feature dimension is frequently attended to in attentional interactions across multiple time steps, its importance score is higher, indicating that this dimension is more valuable for smoke timing discrimination. The core output of multi-head self-attention is the attention score matrix. ( Indicates the first The importance of each attention head (h attention heads) is essentially a weighted measure of temporal dependence. This section will derive the importance of feature dimensions step by step from this weight, requiring averaging across the multi-head and temporal dimensions. The global attention matrix, which integrates multi-head information, is obtained by averaging the score matrices of the h attention heads element-wise. :

[0078] This step averages the score matrices of multiple attention heads, eliminating bias from a single subspace and obtaining a more stable measure of temporal dependency. For the global attention matrix... The temporal importance score for each feature dimension is obtained by averaging over the time dimension L. :

[0079] in For indicator functions, This represents the original dimension to which the feature at time step j belongs. That is, the average contribution of the c-th dimension feature throughout the entire time window.

[0080] Next is the ranking of feature importance. The C feature dimensions are ranked according to... Sort from highest to lowest to obtain the sorted index. ,in Before selection Each index constitutes a subset of the indexes for the optimal feature. ,in Take in this article By optimal index subset For the original time series feature sequence Dimensional clipping is performed to obtain the filtered feature sequences. This filtering process is performed only once during the system design phase; during online runtime, it is directly performed based on the optimal index subset. Extract this Dimensional features.

[0081] Based on the above process, for each trajectory and the specified time window length The output shape of this module is Feature sequence matrix ,Right now:

[0082] in This indicates that the standardized feature vector is taken at the selected index. On Subvectors of dimension. This matrix. It will be used as input to the "Time Series Modeling and Decision Module".

[0083] (3) Time series modeling and decision-making module This module is responsible for performing in-depth analysis and final classification of the optimized temporal feature sequences output by the feature extraction and processing module to determine whether the tracked target trajectory is real smoke, thereby achieving highly reliable false detection filtering.

[0084] This module aims to model the dynamic behavior patterns of suspected smoke targets over time. By learning the unique temporal features of smoke, such as diffusion and irregular movement, it distinguishes smoke from stationary or regularly moving interfering objects. To effectively model complex dependencies in long sequences, the temporal processing network of this invention avoids the gradient vanishing and long-range dependency capture difficulties that may exist in traditional recurrent neural networks (RNNs). Instead, it adopts a Transformer architecture based on self-attention and an encoder-only architecture as the core temporal modeling and decision-making architecture. The overall model structure diagram is shown below. Figure 5 As shown, the structure diagram of the timing coding module used is as follows: Figure 6 As shown, this architecture can compute the correlation between features at any two time steps in a sequence in parallel through a multi-head self-attention mechanism, thereby globally and dynamically capturing the temporal patterns of smoke evolution. Unlike the classic Seq2Seq Transformer, the Encoder-Only architecture removes the decoder, resulting in a more streamlined structure specifically designed for deep understanding and representation of sequences, making it well-suited for current sequence classification tasks.

[0085] This module takes as input the shape of the output from the feature extraction and processing module. Feature sequence matrix Output a two-dimensional probability vector ,in This indicates the probability that the feature extracted at this moment on the trajectory is a false detection. This represents the probability that it is real smoke, and The final decision can be made through comparison. The result is obtained by comparing it with a preset threshold (such as 0.5).

[0086] The steps for processing the input feature sequence in this model are as follows: To preserve the temporal order information of the sequence, in the input feature sequence Add a learnable classification token vector ([CLS] Token) before it. and with learnable positional encoding The summation forms the input sequence of the Transformer encoder. ,Right now:

[0087] Where; indicates concatenation along the sequence dimension, and the [CLS] marker is at the beginning of the sequence. Given a module consisting of N identical Transformer encoder layers stacked together, after N layers of encoding, an enhanced contextual feature sequence is obtained. ,in The final layer's [CLS] token representation aggregates global contextual information from the entire sequence through a self-attention mechanism.

[0088] To construct a more comprehensive sequence-level representation, this application employs a dual-path feature fusion strategy, extracting information from both global semantic and local statistical perspectives. The structure diagram of the "temporal pooling and classification module" used in this strategy is shown below. Figure 7 As shown.

[0089] Directly retrieve the [CLS] Token vector from the final layer. This vector is used to represent global contextual semantic information. It encapsulates complex long-range dependencies between sequence elements, representing the overall dynamic evolution pattern of the target. This applies to all time-step features in the final layer, excluding the [CLS]Token. Global Average Pooling (GAP) is used to represent local statistical information, and its mean vector is calculated. :

[0090] This vector describes the average strength and distribution of the sequence across all feature dimensions, providing a measure of local feature stability. Subsequently, these two representations are concatenated to form the final fused feature vector. :

[0091] This fusion strategy draws on design principles from advanced vision-language alignment models, allowing the model to simultaneously utilize temporal contextual semantics and statistical characteristics. The fused feature vector g is input into a classification head consisting of a fully connected (Linear) layer and a Softmax function to obtain the final classification probability distribution:

[0092] in, and These are the learnable parameters for the classification head. Then, using the labeled temporal feature sequence dataset constructed through the automated process of this invention, end-to-end supervised training is performed using the Cross-Entropy Loss function as the loss function. After training, this module can make temporal judgments.

[0093] Example 1: Feature Extraction and Misjudgment Elimination Pipeline Integrating Target Features and Feature Importance Screening This embodiment details the core data processing flow used in this application to achieve high-precision smoke recognition and false alarm elimination. Addressing the problem of high false alarm rates in existing single-frame smoke detection methods under complex environments, this embodiment designs a complete "detection-tracking-feature extraction-temporal modeling" pipeline.

[0094] First, real-time object detection and multi-object tracking are performed on the input video stream, generating a spatiotemporal trajectory with a unique identifier (ID) for each suspected smoke region. This forms the basis for subsequent temporal analysis, ensuring that feature extraction is aligned with the same target in both time and space. Next, for each frame in the trajectory, two types of complementary features are extracted in parallel from the corresponding image region: 1) visual semantic features are extracted from the feature pyramid of the object detection model (e.g., YOLOv8); 2) frequency domain structural features are extracted by performing Fast Fourier Transform (FFT) and Discrete Cosine Transform (DCT) on the region image. These features are then standardized and concatenated into a high-dimensional feature vector. To automatically select the most effective subset for distinguishing dynamic smoke patterns from a large number of features, this embodiment further designs a sequence-level feature importance analysis mechanism. Specifically, a temporal feature sequence is constructed for a large number of labeled smoke and false-detection trajectory samples, and various statistics (such as mean, standard deviation, rate of change, etc.) of each original feature on the sequence are calculated to form ultra-high-dimensional sequence statistical features. The importance of these statistical features is evaluated using a random forest model, and then aggregated back into the original feature dimensions. Finally, based on the importance scores, the top N (e.g., 128) most discriminative feature dimensions are selected to form the optimized feature set for subsequent time series modeling.

[0095] Example 2: Encoder-Only Transformer Temporal Modeling and Decision-Making Method Based on Dual-Path Feature Fusion This embodiment specifically illustrates how to use a deep learning model to perform end-to-end classification decisions on the optimized temporal feature sequences generated in Embodiment 1. The core of this embodiment lies in adopting and improving the Encoder-Only Transformer architecture and introducing a dual-path feature fusion strategy.

[0096] In specific implementation, the temporal feature sequence from Example 1 is used as input. A learnable classification label is added to the beginning of the sequence and added to the positional encoding before being fed into a multi-layer Transformer encoder. The encoder uses a self-attention mechanism to allow features from any two time steps in the sequence to interact, thereby deeply mining the temporal dependency patterns of smoke. After Transformer encoding, global contextual representation and local statistical representation are extracted to obtain a fused feature vector. Finally, this fused feature vector is input into a lightweight fully connected classification head, which outputs the probability that this sequence of the trajectory is "smoke" or "false detection". By setting a decision threshold, real-time, automatic alarm and filtering can be achieved.

[0097] Example 3: Automated Method for Constructing Time-Series Training Datasets This embodiment provides an efficient and automated method for constructing a training dataset for a temporal smoke detection model. This method significantly reduces data annotation costs and ensures data consistency with the task. The specific implementation steps are as follows: First, a large number of videos containing real highway scenes are collected. These videos only require coarse video-level annotation (e.g., if the video contains a smoke event, the video name contains "smoke"; if it does not contain a smoke event, the video name does not contain "smoke"), eliminating the need for manual frame-by-frame annotation of image-level bounding boxes. Then, the "detection-tracking-feature extraction" process refined in Embodiment 1 is used to perform offline batch processing on these videos. The system automatically generates a complete temporal trajectory for each tracked and detected suspected smoke target (identified as a smoke target by the YOLO algorithm) in the video, and extracts optimized features for each frame along the trajectory (as described in Embodiment 1), thereby automatically generating a feature for each trajectory at each time step after the Lth time step. The feature sequence samples are then generated. The category label for each time series sample is automatically inherited from the coarse annotation of its source video. This method allows for the rapid construction of a time-series feature dataset with extremely low annotation costs. This dataset can be directly used to train the time-series Transformer classification model in Example 2, enabling the model to learn to distinguish complex time-series patterns of smoke and various interferences from massive amounts of real-world scene data, thereby significantly improving the model's generalization ability and practical deployment performance.

[0098] (4) Advantages and beneficial effects In summary, to overcome the limitations of single-frame image detection, this application creatively proposes a complete "detection-tracking-temporal analysis" process, deeply mining and utilizing the unique dynamic behaviors of smoke in the temporal dimension, such as irregular diffusion and rising. The feature extraction and processing module in this application integrates multi-level visual features and frequency domain statistical properties of the target, constructing a rich 256-dimensional original feature set. By introducing a sequence-level feature importance analysis and filtering mechanism based on Transformer self-attention, the system can automatically identify the most effective feature subset for temporal discrimination from a large number of features, achieving automated dimensionality reduction and optimization of feature representation, providing more discriminative input for the backend temporal model. This application can effectively distinguish real smoke from highly similar static or regularly moving interference objects such as nighttime vehicle headlight halos, rainy road surface reflections, and lens water droplets, thereby significantly filtering false alarms caused by misdetection at the source. This solves the core pain point of high false alarm rates in existing technologies deployed in actual highway scenarios, greatly improving the practicality and reliability of smoke detection and alarm monitoring systems. A schematic diagram is shown below. Figure 8 As shown.

[0099] This application also provides a video smoke detection system based on temporal dynamic features, which can implement the above method. The system includes: The target detection and trajectory generation module is used to perform target detection on image frames in the input video stream, obtain the suspected smoke region in each frame, and perform target tracking on the suspected smoke region in multiple consecutive frames, generating a unique spatiotemporal trajectory for each suspected smoke region. The feature extraction and processing module is used to extract the image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory for each spatiotemporal trajectory, and construct a feature sequence based on the image features; The temporal modeling and decision module is used to input the feature sequence into a pre-trained temporal classification model and output the category corresponding to the spatiotemporal trajectory through the temporal classification model. The category includes smoke category or false detection category.

[0100] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0101] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0102] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0103] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0104] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0105] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0106] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0107] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0108] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0109] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0110] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0113] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0114] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0116] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video smoke detection method based on temporal dynamic features, characterized in that, The method includes the following steps: Perform target detection on image frames in the input video stream to obtain suspected smoke regions in each frame; Target tracking is performed on the suspected smoke regions in multiple consecutive frames of images, and a unique spatiotemporal trajectory is generated for each suspected smoke region; For each of the aforementioned spatiotemporal trajectories, image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory are extracted, and a feature sequence is constructed based on the image features; The feature sequence is input into a pre-trained temporal classification model, and the temporal classification model outputs the category corresponding to the spatiotemporal trajectory, which includes smoke category or false detection category.

2. The method according to claim 1, characterized in that, The step of extracting image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory, and constructing a feature sequence based on the image features, includes: For each frame in the spatiotemporal trajectory, multiple features are extracted from the image region corresponding to the suspected smoke area, and the multiple features are concatenated to form a single-frame original feature vector. The original feature vector of the single frame is standardized to obtain the standardized single frame feature vector; The standardized single-frame feature vectors of L consecutive frames in the spatiotemporal trajectory are arranged in chronological order to construct the feature sequence, where L is the preset time window length.

3. The method according to claim 2, characterized in that, The multiple features include visual features and frequency domain features; The visual features include: hierarchical features extracted from the feature pyramid of the target detection model, and channel importance features determined based on the average activation intensity of each channel in the feature pyramid within the suspected smoke area; The frequency domain features include: frequency domain statistical features extracted by performing Fast Fourier Transform on the image region, frequency domain structural features extracted by performing Discrete Cosine Transform, and texture roughness features extracted based on frequency domain gradient.

4. The method according to claim 2 or 3, characterized in that, It also includes a feature importance screening step: A time-series feature importance analysis model based on Transformer self-attention mechanism is used to process a training dataset consisting of multiple feature sequences, learn the contribution of each feature dimension in the time-series evolution, and select a preset number of discriminative feature dimensions from all feature dimensions based on the contribution. The construction of the feature sequence specifically involves: extracting the corresponding feature values ​​from the standardized single-frame feature vector based on the selected discriminative feature dimensions, and constructing a simplified feature sequence.

5. The method according to claim 4, characterized in that, The processing steps of the temporal feature importance analysis model based on the Transformer self-attention mechanism include: The input feature sequence is passed through a multi-head self-attention layer, and the multi-head attention score matrix is ​​calculated. The multi-head attention score matrix is ​​averaged element-wise to obtain a global attention matrix that integrates multi-head information. The global attention matrix is ​​averaged over the time dimension to obtain the temporal importance score for each feature dimension; The feature dimensions are sorted according to the temporal importance score, and a preset number of feature dimensions with the highest ranking are selected as the discriminative feature dimensions.

6. The method according to claim 1, characterized in that, The time-series classification model is a classification model built on the Encoder-OnlyTransformer architecture; The step of inputting the feature sequence into a pre-trained temporal classification model and outputting the category corresponding to the spatiotemporal trajectory through the temporal classification model includes: A learnable classification label is added before the feature sequence and then added to the positional encoding before being input into a multi-layer Transformer encoder. Obtain the enhanced feature sequence output by the multi-layer Transformer encoder, the enhanced feature sequence including the enhanced classification label vector and the enhanced feature vectors at each time step; The enhanced classification label vector is fused with the mean vector obtained by global average pooling of the enhanced feature vectors at each time step to obtain the fused feature vector. The fused feature vector is input into the classification head to obtain the probability that the spatiotemporal trajectory belongs to the smoke category or the false detection category.

7. A video smoke detection system based on temporal dynamic features, characterized in that, include: The target detection and trajectory generation module is used to perform target detection on image frames in the input video stream, obtain the suspected smoke region in each frame, and perform target tracking on the suspected smoke region in multiple consecutive frames, generating a unique spatiotemporal trajectory for each suspected smoke region. The feature extraction and processing module is used to extract the image features of the suspected smoke region corresponding to each image frame in the spatiotemporal trajectory for each spatiotemporal trajectory, and construct a feature sequence based on the image features; The temporal modeling and decision module is used to input the feature sequence into a pre-trained temporal classification model and output the category corresponding to the spatiotemporal trajectory through the temporal classification model. The category includes smoke category or false detection category.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images