Day and night dual mode firework detection method and device

Abstract

The application discloses a kind of day and night dual mode fireworks detection method and device, the method includes: according to ambient light condition, determine acquisition mode;If acquisition mode is daytime mode, the color image sequence collected by visible light camera is input to first fireworks detection model, to judge whether there is fire event;If acquisition mode is night mode, the gray image sequence collected by near-infrared light supplement camera is input to first fireworks detection model, the second detection result obtained is input to physical perception convolution timing model, if the adaptive fractional fourier transform feature of the luminance feature sequence of the suspected fireworks area of each frame gray image in the third detection result obtained meets the fireworks spectrum mode and the edge jitter meets the fireworks dynamic characteristics, determine that there is fire event;Physical perception convolution timing model generates the timing correction coefficient of each frame gray image to screen gray image.The application can adapt to all-weather environment, with high robustness and low false alarm rate.

Description

Technical Field

[0001] This invention relates to the field of safety production data processing and monitoring technology, and in particular to a day and night dual-mode smoke and fire detection method and device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Existing video-based smoke detection algorithms have many shortcomings in complex environments.

[0004] First, traditional visible light smoke detection algorithms rely on the color and motion characteristics of smoke. Under low-light conditions at night, the detection performance deteriorates sharply due to a significant drop in image quality. Some systems use infrared or thermal imaging equipment to deal with nighttime scenes, but such specialized equipment is expensive and complex to deploy and maintain, limiting its widespread application.

[0005] Secondly, in nighttime surveillance, visible light cameras often need to rely on infrared supplementary lighting for imaging. The resulting grayscale images lack color characteristics and are prone to misinterpreting bright light sources such as car headlights and fluorescent lights as smoke and fire, causing false alarms.

[0006] Finally, deep learning-based smoke detection models rely on large-scale, high-quality data for training. However, images of nighttime fires are scarce in reality, and publicly available datasets are insufficient, resulting in weak generalization ability of existing models in day-night transition scenarios. Therefore, there is an urgent need to develop a new smoke detection method that can adapt to all-weather environments and possesses high robustness and a low false alarm rate. Summary of the Invention

[0007] This invention provides a day-night dual-mode smoke detection method that can adapt to all-weather environments, has high robustness and low false alarm rate, including:

[0008] Determine the data acquisition mode based on the current ambient lighting conditions;

[0009] If the acquisition mode is daytime mode, the color image sequence acquired by the visible light camera is input into the first smoke detection model to obtain the first detection result. Based on the first detection result, it is determined whether there is a fire event. The first smoke detection model is a convolutional neural network structure.

[0010] If the acquisition mode is night mode, the gray image sequence acquired by the near-infrared supplementary light camera is input into the first smoke detection model to obtain the second detection result. The second detection result is then input into the physical perception convolutional temporal model to obtain the third detection result. If the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected smoke area in each frame of the gray image in the third detection result conform to the smoke spectrum pattern and the edge jitter satisfies the dynamic features of smoke, a fire event is determined to exist. The second detection result includes the bounding box and confidence score of each frame of gray image. The physical perception convolutional temporal model is used to generate temporal correction coefficients for each frame of gray image through gating modulation. Gray images are then filtered through the temporal correction coefficients to obtain the third detection result.

[0011] This invention provides a day / night dual-mode smoke detection device that can adapt to all-weather environments, has high robustness and low false alarm rate. The device includes:

[0012] The acquisition mode determination module is used to determine the acquisition mode based on the current ambient lighting conditions.

[0013] The daytime detection module is used to input the color image sequence captured by the visible light camera into the first smoke detection model if the acquisition mode is daytime mode, obtain the first detection result, and determine whether there is a fire event based on the first detection result. The first smoke detection model is a convolutional neural network structure.

[0014] The nighttime detection module is used to input the gray image sequence captured by the near-infrared supplementary light camera into the first smoke detection model to obtain a second detection result if the acquisition mode is nighttime mode. The second detection result is then input into the physical perception convolutional temporal model to obtain a third detection result. If the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected smoke area in each frame of the gray image in the third detection result conform to the smoke spectrum pattern and the edge jitter satisfies the dynamic features of smoke, a fire event is determined to exist. The second detection result includes the bounding box and confidence score of each frame of gray image. The physical perception convolutional temporal model is used to generate temporal correction coefficients for each frame of gray image through gating modulation. The gray images are then filtered through the temporal correction coefficients to obtain the third detection result.

[0015] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described day and night dual-mode smoke detection method.

[0016] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described day-night dual-mode smoke detection method.

[0017] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described day and night dual-mode smoke detection method.

[0018] In this embodiment of the invention, the system adapts to different lighting environments to achieve accurate and comprehensive collection of smoke and fire data throughout the day and night, ensuring the validity of basic data for smoke and fire detection at different times. A first smoke and fire detection model with a convolutional neural network structure is used to improve the extraction and recognition efficiency of smoke and fire features in daytime color image sequences, ensuring the accuracy of daytime fire event judgment. In the intermittent mode, the first smoke and fire detection model is combined with a physical perception convolutional temporal model. Images are filtered through temporal correction coefficients, improving the targeting of nighttime grayscale image sequence processing and reducing interference from invalid images on the detection results. Through dual judgment of adaptive fractional Fourier transform features and edge jitter, the spectral and dynamic features of suspected smoke and fire areas at night are accurately captured, improving the distinction between nighttime smoke and fire and other bright nighttime interference targets, reducing the false alarm and false negative rates of nighttime smoke and fire detection. The overall technical process enables adaptive adjustment of smoke and fire detection to changes in lighting, enhancing the robustness and reliability of the smoke and fire detection system in complex environments at different times. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0020] Figure 1 This is a flowchart of the day-night dual-mode smoke detection method in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram illustrating how an incandescent lamp might be mistakenly detected as fireworks by the first detection model in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the principle of model training in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating the principle of generating composite images of fireworks videos in an embodiment of the present invention.

[0024] Figure 5 This is a detailed flowchart of the day and night dual-mode smoke detection method in this embodiment of the invention;

[0025] Figure 6 This is a schematic diagram of the day and night dual-mode smoke detection device in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] Figure 1 This is a flowchart of a day / night dual-mode fireworks detection method according to an embodiment of the present invention. The method is applied to a first node in an asynchronous distributed system, where the first node can be any node in the asynchronous distributed system. The method includes:

[0029] Step 101: Determine the acquisition mode based on the current ambient lighting conditions;

[0030] Step 102: If the acquisition mode is daytime mode, input the color image sequence acquired by the visible light camera into the first smoke detection model to obtain the first detection result. Based on the first detection result, determine whether there is a fire event. The first smoke detection model is a convolutional neural network structure.

[0031] Step 103: If the acquisition mode is night mode, input the gray image sequence acquired by the near-infrared supplementary light camera into the first smoke detection model to obtain the second detection result. Input the second detection result into the physical perception convolutional temporal model to obtain the third detection result. If the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected smoke area in each frame of the gray image in the third detection result conform to the smoke spectrum pattern and the edge jitter satisfies the dynamic features of smoke, it is determined that a fire event exists. The second detection result includes the bounding box and confidence score of each frame of gray image. The physical perception convolutional temporal model is used to generate temporal correction coefficients for each frame of gray image through gating modulation. The gray images are filtered through the temporal correction coefficients to obtain the third detection result.

[0032] Each step is described in detail below.

[0033] In step 101, the acquisition mode is determined based on the current ambient lighting conditions;

[0034] In this embodiment of the invention, when the ambient light is higher than a set threshold, a visible light camera is used; otherwise, a near-infrared supplementary light camera is used to ensure that clear images can be output under different lighting conditions.

[0035] In step 102, if the acquisition mode is daytime mode, the color image sequence acquired by the visible light camera is input into the first smoke detection model to obtain the first detection result. Based on the first detection result, it is determined whether there is a fire event. The first smoke detection model is a convolutional neural network structure.

[0036] In this embodiment of the invention, the fireworks include flames and smoke. The brightness variations of the fireworks in the grayscale image sequence captured by the near-infrared supplementary lighting camera are light intensity perturbations rather than thermal radiation.

[0037] In one embodiment, the first detection result includes the bounding box and confidence level of each frame of color image;

[0038] Based on the initial detection results of all color images, determine whether a fire event exists, including:

[0039] If the confidence level of a consecutive preset number of color images exceeds a first threshold, and the shape and size of the bounding boxes of the consecutive preset number of color images meet preset requirements, a fire event is determined to exist.

[0040] In this embodiment of the invention, during the daytime mode, due to relatively little environmental interference, the presence of smoke is primarily determined based on a first smoke detection model using a YOLO architecture. Subsequently, the presence of a fire event is determined based on the smoke. Upon confirmation of a fire event, an alarm signal and related actions (such as video recording and alarm sound) can be triggered. Before inputting the color image sequence into the first smoke detection model, all frames of color images can be scaled to the preset input size of the YOLO network (e.g., 640×640) to avoid feature extraction bias caused by size differences.

[0041] In one embodiment, the first fireworks detection model includes a feature extraction network, a detection head, and a non-maximum suppression module:

[0042] The feature extraction network includes a CSPDarknet backbone network and a neck feature fusion module (PANet), used to extract multi-scale spatial features (such as the brightness distribution and shape contour of fireworks) from color images (or grayscale images).

[0043] The detection head includes a bounding box regression branch, a confidence prediction branch, and a category branch, which are used to output the following content for each candidate region:

[0044] bounding box, using express: The coordinates of the center point of the bounding box. These are the width and height, respectively.

[0045] Confidence The value range is [0,1], which represents the single-frame reliability of the candidate region being fireworks;

[0046] Category (c): 0 indicates no fireworks, 1 indicates fireworks.

[0047] The Non-Maximum Suppression (NMS) module is used to filter the bounding boxes of candidate regions output in each frame, removing redundant boxes with an overlap (IOU) exceeding a threshold (e.g., 0.5), and retaining the bounding boxes of candidate regions with the highest confidence.

[0048] The first fireworks detection model outputs "bounding box + confidence score + category" for each frame of color or grayscale image (e.g., frame t output). , c).

[0049] In this embodiment of the invention, the near-infrared supplementary lighting camera is an infrared camera under supplementary lighting. Although the gray image sequence lacks color, it can present the brightness distribution of objects in the scene.

[0050] In one embodiment, the second detection result includes the bounding box and confidence level of each grayscale image frame;

[0051] The physical-aware convolutional temporal model includes:

[0052] The stitching module is used to crop the suspected ROI of each grayscale image based on the bounding box of each grayscale image and construct a brightness feature sequence of consecutive frames.

[0053] The calibration module is used to generate temporal correction coefficients for each frame of grayscale image based on the brightness feature sequence and through gated modulation.

[0054] The fusion module is used to perform weighted fusion of the confidence score and temporal correction coefficient of each grayscale image to obtain the final confidence score of each grayscale image.

[0055] The filtering module is used to filter the bounding boxes based on the final confidence scores of all frames of grayscale images, and obtain the filtered bounding boxes and their final confidence scores to form the third detection result.

[0056] In this embodiment of the invention, the Phy-ConvLSTM model combines the spatial feature extraction capability of convolutional neural networks with the temporal series modeling advantages of LSTM, effectively handling long-term dependencies in image sequences. Specifically, it transmits temporal information through hidden states and ultimately outputs the hidden state at the last moment. This hidden state encodes the dynamic pattern of the entire image sequence, thereby distinguishing real fireworks from static interference (such as red objects or fixed shadows), improving classification accuracy.

[0057] Before processing the grayscale image sequence, all frames of grayscale images can be scaled to the preset input size of the first fireworks detection model (e.g., 640×640) to avoid feature extraction deviations caused by size differences. Perform the operation (pixel value - mean grayscale) / standard deviation of grayscale on each frame of grayscale image to reduce the impact of light intensity fluctuations on the first fireworks detection model. Use Gaussian filtering or median filtering to remove salt-and-pepper noise in the grayscale images and retain the brightness and edge information of the fireworks area.

[0058] The stitching module can be based on the bounding box of each grayscale image ( The suspected fireworks area (ROI) is cropped from the corresponding grayscale image to ensure that subsequent temporal modeling focuses only on the target area and reduces background interference; then, temporal feature alignment is performed to extract the brightness feature sequence of N consecutive frames from the ROI. ( (where is the average brightness of the ROI in frame t).

[0059] In one embodiment, the calibration module includes:

[0060] The physical frequency feature extraction module is used to generate a physical consistency score for fireworks based on the brightness feature sequence.

[0061] The gated modulation module is used to obtain a physically constrained basic input gate and a basic forget gate based on convolution operations, and to perform physical constraint modulation on the basic input gate and the basic forget gate using the pyrotechnic physical consistency score to obtain a modulated input gate and a modulated forget gate.

[0062] The cell state update module is used to update the cell state of the current frame based on the modulation input gate, the modulation forget gate, the hidden state of the previous frame, the brightness feature matrix of the current frame, and the cell state of the previous frame.

[0063] The hidden state output module is used to calculate the output gate based on the brightness feature matrix of the current frame and the hidden state of the previous frame, and to calculate the hidden state of the current frame based on the output gate and the cell state of the current frame.

[0064] The mapping module is used to map the hidden state of the current frame to the timing correction coefficient of the current frame.

[0065] In this embodiment of the invention, the collected brightness feature sequence is a continuous sequence of multiple frames, used to represent brightness changes.

[0066] The physical consistency score of fireworks can provide a differentiable physical constraint for subsequent gating modulation. The brightness features of fireworks can be extracted from two consecutive frames of an image sequence, and the physical consistency score of fireworks can be calculated.

[0067] In one embodiment, the physical frequency feature extraction module is used for:

[0068] Calculate the brightness feature change matrix based on the brightness feature matrix of the current frame and the brightness feature matrix of the previous frame in the brightness feature sequence.

[0069] Calculate the spectrum of brightness feature changes based on the brightness feature change matrix;

[0070] Calculate the dominant frequency component of the spectrum of brightness characteristic changes;

[0071] The physical consistency score of the fireworks is calculated based on the main frequency component and the preset frequency band range of the fireworks flashing.

[0072] In this embodiment of the invention, the brightness feature change matrix can be represented as:

[0073]

[0074] in, This is the brightness feature variation matrix. This is the brightness feature matrix of the current frame. This is the brightness feature matrix of the previous frame.

[0075] Spectrum of brightness feature changes The brightness feature change matrix can be obtained by performing a windowed FFT or DFT:

[0076]

[0077] The main frequency component can be represented as:

[0078]

[0079] When calculating the physical consistency score of fireworks based on the dominant frequency component and the preset frequency band range of fireworks flashing, it can be determined whether the dominant frequency component is within the preset frequency band range of fireworks flashing. Within, for example, 5–15 Hz, and with a differentiable sigmoid hard constraint:

[0080]

[0081] in, The physical consistency of the fireworks is scored, with a score closer to 1 indicating that it is more like real fireworks and a score closer to 0 indicating that it is more like periodic light flashes or noise. It can be used as a core physical constraint signal and directly transmitted to the subsequent gating and modulation module to adjust the weights of the input gate and the forget gate.

[0082] The core function of the gated modulation module is based on a standard gated structure, incorporating the output of the physical frequency feature extraction module. The system selectively receives and selectively forgets information streams to suppress non-smoke interference signals. Inputs include the brightness feature matrix of the current frame and the hidden state of the previous frame. (Historical time series characteristics); Physical consistency score (From the physical frequency feature extraction module), output modulation input gate and modulating the forgetting gate (Weights with physical constraints), the calculation process is as follows:

[0083] First, obtain the basic gating values ​​without physical constraints through convolution operations:

[0084] Basic input gates:

[0085] The basic input gate controls whether the brightness feature matrix of the current frame enters the cell state. and As weight;

[0086] Basic forgetting gate:

[0087] The basic forget gate controls whether the cell state of the previous frame is retained. and As weight;

[0088] Modulation input gate:

[0089] These are learnable parameters. If... If the value is close to 1 (corresponding to the frequency of fireworks), then the input gate weights are increased, allowing the model to prioritize receiving the frame features of the current frame; if... If the frequency is close to 0 (non-smoke frequency), the input gate is suppressed to reduce the input of interference signals;

[0090] Modulating the forgetting gate:

[0091] For learnable parameters, if Approaching 1 reduces the weight of the forget gate, allowing the model to retain historical fireworks features; if Approaching 0 amplifies the forget gate weight, preventing the accumulation of interference signals.

[0092] The modulation input gate and modulation forget gate are directly fed into the cell state update module as the core control signals for cell state update.

[0093] The core function of the cell state update module is to combine the modulation input gate, modulation forget gate, the hidden state of the previous frame, the brightness feature matrix of the current frame, and the cell state of the previous frame. (Historically accumulated temporal characteristics) Update the cell state of the current frame. This process accumulates and filters brightness features, preserving the dynamic characteristics of fireworks that conform to physical laws. The inputs include a modulation input gate and a modulation forget gate, the brightness feature matrix of the current frame, the hidden state of the previous frame, and the cell state of the previous frame. (Historically accumulated temporal characteristics). Output the updated cell state for the current time frame. (The set of temporal features that incorporates physical constraints), the calculation formula is:

[0094]

[0095] Cell state renewal utilizes Filter the cell state of the previous time frame When the physical properties of fireworks are met, more historical characteristics are preserved; when they are not, most historical interference is forgotten. Filtering latent features of the current frame When the characteristics of the fireworks conform to the physical properties of the fireworks, the features of the current frame are incorporated first; when they do not conform, the interference features of the current frame are weakened.

[0096] The core function of the hidden state output module is to output the cell state of the current frame. Perform activation and filtering, and output the hidden state of the current frame. This provides temporal dynamic characteristics for subsequent fusion decision-making in fireworks detection.

[0097] The hidden state output module outputs the output based on the luminance feature matrix of the current frame and the hidden state of the previous frame. Calculate the output gate, and then output the hidden state of the current frame based on the output gate and the cell state of the current frame. (The final dynamic characteristics, which integrate physical constraints, spatial features, and temporal features), the calculation formula for the output gate is:

[0098]

[0099] The formula for generating the hidden state of the current frame is:

[0100]

[0101] The output gate filters out effective features from the cell state of the current frame, which are then activated by tanh to obtain... ;

[0102] On the one hand, it serves as the hidden state of the current frame (the temporal features of the current frame), which is fused with the extracted spatial features to correct the confidence level of suspected fireworks areas; on the other hand, it serves as... By passing in the next time frame, the continuous transmission and accumulation of temporal features can be achieved.

[0103] right Perform global average pooling + fully connected layer + Sigmoid activation, and then... High-dimensional temporal feature mapping to single-valued temporal correction coefficients (Value range [0,1]).

[0104] Final confidence level During the calculation, a weighted fusion strategy is used, as shown in the following formula:

[0105]

[0106] Final confidence level Learnable weights can also be introduced during the calculation. The announcement is as follows:

[0107]

[0108] The above formula can ensure the joint discrimination of spatial features and temporal features.

[0109] In the filtering module, you can retain... ( The final bounding box of the grayscale image is determined by a preset threshold (e.g., 0.6), and those bounding boxes that exceed the grayscale image range or are of abnormal size are identified, resulting in the filtered final bounding boxes. and final confidence level .

[0110] In one embodiment, it further includes:

[0111] Deep features of the brightness feature sequence of the suspected fireworks area in each frame of the gray image in the second detection result are extracted by a one-dimensional convolutional temporal coding network to obtain the temporal feature vector;

[0112] Calculate the optimal order of the fractional Fourier transform based on the temporal feature vector;

[0113] Based on the optimal order, an adaptive fractional Fourier transform is performed on the brightness feature sequence to obtain the frequency domain signal;

[0114] The adaptive fractional Fourier transform features are extracted from the frequency domain signal. These features include fractional energy, fractional spectral entropy, spectral bandwidth, and main peak value.

[0115] The adaptive fractional Fourier transform features are compared with the corresponding threshold range to determine whether they conform to the fireworks spectrum pattern.

[0116] In this embodiment of the invention, deep features of the brightness feature sequence are extracted by a one-dimensional convolutional temporal coding network (Conv1D network). Before extracting the deep features, the brightness feature sequence of the suspected fireworks area of ​​each frame of gray image in the second detection result in the most recent period can be preprocessed, including baseline correction, normalization and noise filtering, to provide a basis for adaptive order generation.

[0117] Input the luminance feature sequence (or the preprocessed luminance feature sequence) into the Conv1D network, and output the temporal feature vector. (Dimensions: (C, L), where C is the number of convolution kernels and L is the sequence length, where, The global average pooling is compressed into a one-dimensional normalized vector (e.g., dimension 1×32). The normalized vector ensures the stability of subsequent order generation.

[0118] Based on time-series feature vectors Calculating the optimal order of the fractional Fourier transform maximizes the separation of the characteristic peaks of fireworks from interference sources (such as LEDs) in the frequency domain.

[0119] Substituting the order generation formula, and limiting the output range through a fully connected layer and Sigmoid activation, the optimal order is obtained. :

[0120]

[0121] This is the weight matrix for the fully connected layer (dimension: 1×C, where C is the dimension of h).

[0122] This is a bias term (scalar);

[0123] For the Sigmoid activation function, ensure (Corresponding to the order range, order 0 is the original signal, and order 1 is the standard FFT).

[0124] like If the value is close to 0 or 1 (which is likely to overlap with the standard FFT / original signal and lose its adaptive advantage), it should be adjusted to a preset range value (such as 0.2~0.8) to ensure that the order can map the fireworks features to the optimal separation frequency domain.

[0125] Based on the optimal order Perform an adaptive fractional Fourier transform (FrFT) on the brightness feature sequence (or the preprocessed brightness feature sequence if preprocessed) to map the time-domain brightness signal to the adaptive frequency domain, highlighting the fireworks features and suppressing interference, to obtain the frequency domain signal:

[0126]

[0127] The above formula is adjustable By altering the time-domain to frequency-domain mapping angle, the flicker characteristic peak of the fireworks appears at the most easily distinguishable position in the frequency domain, while suppressing the characteristics of interference from LEDs and other sources, thus enhancing the frequency domain signal. Let be a complex sequence, where u is the frequency domain sampling point, and the number of u is the same as that of T.

[0128] From frequency domain signals Four core features—fractional energy, spectral entropy, spectral bandwidth, and main peak value—are extracted and used as the basis for verifying the spectral patterns of fireworks.

[0129] The formula for calculating the fractional energy E is:

[0130]

[0131] The fractional energy E reflects the total energy of the frequency domain signal. The brightness fluctuation energy of fireworks is within a specific range. The energy of interference sources (such as LEDs) is either too low (static) or too high (high frequency jitter).

[0132] The fractional spectral entropy H is calculated as follows:

[0133] Step 1: Calculate the probability distribution of each sampling point of the frequency domain signal :

[0134]

[0135] Step 2: Calculate the fractional-order spectral entropy H (reflecting the uniformity of the spectral distribution; fireworks have a concentrated spectrum and a moderate entropy value; interference sources have a dispersed / overly concentrated spectrum and a deviating entropy value):

[0136]

[0137] Supplement: If ,but (To avoid logarithmic inconsistencies);

[0138] The calculation process for the spectral bandwidth B is as follows:

[0139] Step 1: Map the frequency domain sampling point u to the actual frequency value f (unit: Hz): ( This refers to the sampling frequency of the luminance sequence, such as a frame rate of 25fps. );

[0140] Step 2: Calculate the frequency corresponding to 5% of the cumulative energy. The frequency corresponding to 95% ;

[0141] Step 3: Spectral bandwidth (The spectrum of fireworks is concentrated in the range of 5~15Hz, and the bandwidth conforms to a specific range).

[0142] The formula for calculating the main peak value P is:

[0143]

[0144] Where P is the maximum amplitude of the frequency domain signal, corresponding to the main frequency component of the firework flash, and the main peak value of the interference source deviates from the firework threshold range.

[0145] The extracted adaptive fractional Fourier transform features Compare each value with the corresponding threshold range to determine whether the requirements are met:

[0146] Single feature verification:

[0147] Fractional energy E: judgment ;

[0148] Fractional spectral entropy H: Judgment ;

[0149] Spectral bandwidth B: Judgment ;

[0150] Main peak value P: Judgment ;

[0151] Multi-feature fusion determination:

[0152] Strict mode: All four adaptive fractional Fourier transform features meet the threshold range, and are judged to conform to the fireworks spectrum mode;

[0153] Relaxed mode (adapted to complex scenarios): at least 3 features must be satisfied, and the main peak value P must be satisfied (core feature) to be judged as conforming to the fireworks spectrum mode;

[0154] If the above conditions are not met, it is determined to be inconsistent with the fireworks spectrum pattern (suspected interference source).

[0155] In one embodiment, it further includes:

[0156] Extract the edge intensity matrix of the suspected fireworks region from any two consecutive grayscale images, and calculate the edge difference matrix based on the Sobel operator;

[0157] The edge difference matrix is ​​filtered according to the preset minimum difference threshold to obtain the filtered edge difference matrix;

[0158] Perform global statistics on the selected edge difference matrix to obtain the edge jitter amount;

[0159] Based on the preset jitter threshold, determine whether the edge jitter amount meets the dynamic characteristics of fireworks.

[0160] In this embodiment of the invention, the suspected smoke / fire area in two consecutive grayscale images is... and Extract the edge intensity matrix (precise contours are not required; only edge distribution and intensity need to be reflected):

[0161] Sobel operator configuration: Employs horizontal (x-direction) and vertical (y-direction) Sobel convolution kernels to extract edge responses in both directions.

[0162] Horizontal Sobel kernel: (Detecting vertical edges);

[0163] Vertical Sobel core: (Detect horizontal edges);

[0164] Edge strength calculation:

[0165] right respectively with Perform convolution operations to obtain the horizontal edge response matrix. and vertical edge response matrix ;

[0166] Calculate the edge intensity matrix (combining edge information from two directions):

[0167]

[0168] Similarly, for Perform the same operation to obtain the edge intensity matrix. ;

[0169] Will and The pixel values ​​are mapped to the [0, 255] interval (for easier subsequent difference calculation), and the normalization formula is:

[0170]

[0171] Similarly, we can obtain .

[0172] By calculating the difference between the edge intensity matrices of two consecutive frames, an edge difference matrix is ​​obtained to capture dynamic changes in edges (firework edges change significantly, with large difference matrix values; static light source edges change little, with difference matrix values ​​close to 0). First, the normalized edge difference matrix is ​​then analyzed. and Perform pixel-by-pixel absolute value difference to obtain the edge difference matrix. :

[0173]

[0174] Where (x, y) are the pixel coordinates of the matrix. , This represents the change in edge intensity at position (x, y) from frame t to frame t+1; a minimum difference threshold is set. (Filtering out minute fluctuations caused by noise), for Perform filtering: If If the value is 0, then set it to 0 to obtain the filtered edge difference matrix. (Highlight effective edge changes).

[0175] Global statistics are performed on the selected edge difference matrix to obtain the edge jitter (a core indicator reflecting the overall edge jitter intensity of the region):

[0176] calculate The sum of all pixel values, i.e., the cumulative edge change:

[0177]

[0178] Divide the cumulative edge change by the total number of pixels in the ROI region, HW (where H is the region height and W is the region width), to obtain the edge jitter amount, EFM.

[0179]

[0180] EFM is the average level of edge intensity change between two consecutive frames. The larger the value, the more severe the edge jitter, and the more likely it is fireworks; the closer the value is to 0, the more stable the edge, and the more likely it is a static light source.

[0181] The calculated edge jitter amount EFM is compared with the preset jitter threshold, and the fireworks judgment result is output:

[0182] If EFM > preset jitter threshold: it means that the edge jitter intensity of the suspected fireworks area meets the dynamic characteristics of fireworks;

[0183] If EFM ≤ preset jitter threshold: it indicates that the edge changes are weak, which is consistent with the characteristics of a static light source and does not meet the dynamic characteristics of fireworks.

[0184] As shown in step 103, a fire event is confirmed only when all three conditions are met simultaneously: "the first detection model detects it as smoke," "the adaptive fractional Fourier transform features of the brightness feature sequence match the smoke spectrum pattern," and "the edge jitter amount meets the dynamic characteristics of smoke." For example, a stationary incandescent light bulb at night might be misdetected as smoke by the first detection model (because of its high brightness and potentially similar shape to a fireball). Figure 2This diagram illustrates how an incandescent lamp might be mistakenly detected as smoke by the first detection model in an embodiment of the invention. However, its brightness spectrum does not match the characteristics of smoke, so the fusion judgment will filter out this detection and not issue an alarm. If a very weak flame appears, frequency domain analysis detects flickering, but the first detection model fails to identify the shape. This multi-level judgment strategy greatly reduces false alarms caused by interference from lights and other sources in nighttime environments.

[0185] This invention also proposes to compensate for the scarcity of real nighttime fireworks samples by using image augmentation technology, and to train a first detection model and a physical perception convolutional temporal model that have both spatial positioning accuracy and temporal dynamic discrimination capabilities, adapting to day and night dual-mode detection scenarios.

[0186] Figure 3 This is a schematic diagram illustrating the principle of model training in an embodiment of the present invention. See [link / reference]. Figure 3 The model training includes training a first detection model, training a physical perception convolutional temporal model, and jointly training the two aforementioned models. In one embodiment, the method further includes:

[0187] Obtain the original real sample set, no-fire night scene images, and smoke template images. The original real sample set includes color image samples and gray image samples of smoke.

[0188] Based on the original real sample set, images of fireless nighttime scenes, and smoke template images, an expanded sample set is obtained through image enhancement and synthesis.

[0189] The first detection model is trained based on the expanded sample set, and the trained first detection model is obtained.

[0190] Based on the trained first detection model, train the physical perception convolutional temporal model to obtain the trained physical perception convolutional temporal model.

[0191] The trained first detection model and the trained physical perception convolutional temporal model are concatenated and then jointly trained.

[0192] In this embodiment of the invention, color image samples can be obtained by splitting daytime RGB fireworks videos into frame sequences, and color image samples are obtained by splitting a small number of near-infrared real fireworks videos into frame sequences; auxiliary materials include images of fireless nighttime scenes (such as roads, construction sites, tunnels) and fireworks template images (grayscale / color).

[0193] Before expanding the sample, all original real sample sets and auxiliary materials can be standardized to provide a unified basis for subsequent expansion operations:

[0194] Image format standardization: Convert all images (RGB / grayscale) to 8-bit single-channel or three-channel images. Grayscale images are standardized to a pixel range of [0,255], while RGB images retain their original channels.

[0195] Size normalization: All images are scaled to a standard size of 640×640 (to adapt to YOLOv8 input), and bilinear interpolation is used to ensure pixel smoothness;

[0196] Video frame splitting: The original fireworks video is split into a continuous frame sequence at a frame rate of 10fps, preserving the temporal correlation between frames for subsequent video synthesis and temporal training.

[0197] In this embodiment of the invention, three methods, namely geometric transformation, pixel transformation, and simulated infrared enhancement, are used to expand the preprocessed original real sample to simulate the shape and lighting changes of the target in the real scene.

[0198] In one embodiment, an expanded sample set is obtained by image enhancement and synthesis based on the original real sample set, images of fireless nighttime scenes, and smoke template images, including:

[0199] A single-frame image from the original real sample set is enhanced to obtain an enhanced image. The enhancement includes at least one of geometric transformation enhancement, geometric transformation enhancement, and infrared-simulated enhancement.

[0200] After adding dynamic brightness perturbation to the fireworks template image, the image is synthesized with a nighttime scene image without fire to obtain a multi-scene fused image, which is then labeled.

[0201] Based on preset text prompts and image constraints, an open-source diffusion model is used to obtain and label images generated by the diffusion model.

[0202] Representative fireworks images are selected from single-frame images in the original real sample set, scene text labels are generated, and a continuous frame sequence is generated by combining the representative fireworks images and scene text labels. Temporal smoothing correction is performed on the continuous frame sequence to obtain a synthetic fireworks video. The synthetic fireworks video is then split into frames to obtain a composite fireworks video image.

[0203] The original real sample set, enhanced images, multi-scene fusion generated images, diffusion model generated images, and fireworks video composite images are merged and deduplicated to form an expanded sample set.

[0204] Geometric transformation enhancement: Randomly perform rotation (±15°), horizontal / vertical flip, and affine transformation (scaling 0.8~1.2 times, cropping ±5°) on a single frame image. After rotation / scaling, the blank areas are filled with the average pixels of the image. Label adaptation: Synchronously transform the bounding box coordinates to ensure that the labels are consistent with the target positions. Generate 3 geometric transformation variants for each class of original samples.

[0205] Pixel transformation enhancement: Randomly perform brightness adjustment (±20%), contrast adjustment (0.8~1.2 times), and Gaussian noise injection (σ=0.01~0.05) on the image. For RGB images, additional saturation adjustment (0.8~1.2 times) is added. The adjusted pixel values ​​are truncated to [0,255] to avoid numerical anomalies. Two pixel transformation variants are generated for each type of geometric transformation variant.

[0206] Pseudo-infrared enhancement (RGB to near-infrared): For color images of fireworks, a weighted formula is used to convert them into pseudo-infrared grayscale images. (α=0.6, β=0.3, γ=0.2, satisfying α+β>γ); the calculation results are truncated to [0,255] and rescaled to highlight the brightness of the fireworks area, and one infrared-like variant is generated for each type of RGB sample; after enhancement, a total of 6 times the number of the original samples are generated, and they are uniformly named "original ID - enhancement type - serial number" (e.g., F_001-rot-bright-01.png).

[0207] In this embodiment of the invention, near-infrared fireworks samples and extreme scene samples are generated specifically through three methods: multi-scene fusion generation, diffusion model generation, and fireworks video synthesis. Dynamic brightness perturbation is added to the fireworks template image to obtain the image after adding dynamic brightness perturbation to the fireworks template image, as shown in the following formula:

[0208]

[0209] in: Fireworks template image; This is the disturbance amplitude coefficient (e.g., 0.1 to 0.3). The flashing frequency of the fireworks; Let be a perturbation phase random variable.

[0210] The image with dynamic brightness perturbation added to the fireworks template image is combined with a nighttime scene image without fireworks to generate a nighttime image with dynamic fireworks. The synthesis formula is as follows:

[0211]

[0212] in, Images of a fireless nighttime scene. This is the image after adding dynamic brightness perturbation to the fireworks template image; and To achieve the desired composition weights, during annotation, the bounding boxes and labels of the composite fireworks areas are dynamically annotated, generating 5000+ single-frame composite samples for nighttime use.

[0213] Diffusion model generation (extreme scene samples): Using an open-source diffusion model (such as Stable Diffusion), input text prompts (such as "faint smoke obscured by thick smoke at night", "smoke at a construction site next to strong sunlight glare", "smoke with infrared supplemental lighting inside a tunnel"); image constraints are: control the generated image to be a 640×640 grayscale image, focus on the near-infrared scene, generate 2000+ single-frame samples of extreme scenes, and manually verify and label them.

[0214] Figure 4 This is a schematic diagram illustrating the principle of fireworks video composite image generation in an embodiment of the present invention. See [link / reference]. Figure 4 During the fireworks video synthesis (temporal sample generation), representative fireworks images are selected from single-frame images, and scene text labels are generated (such as "nighttime infrared supplementary lighting construction site smoke"). The Video Diffusion model is used to combine the representative fireworks images and scene text labels to generate a continuous frame sequence (4-5 seconds in length, 10fps) to ensure the natural dynamics of fireworks flickering and smoke drifting between frames. Then, temporal smoothing correction is performed to eliminate abrupt changes between frames. Finally, the synthesized fireworks video is output. After splitting the frames, multi-frame temporal fireworks video synthesis images are generated, and the time stamp association between frames is preserved.

[0215] Expand the sample set: Divide it into training set, validation set and test set in a ratio of 8:1:1 to ensure that the three sets include day and night, normal / extreme scene samples. The annotation files are divided and their integrity is verified at the same time.

[0216] Based on the expanded sample set, the first detection model is trained, focusing on the spatial appearance feature localization of fireworks:

[0217] First detection model initialization: Load YOLOv8 basic weights, modify the output channels of the detection head (number of categories = 3), and retain the CSPDarknet backbone network and PANet feature fusion module;

[0218] Loss function: YOLO standard loss is used (localization loss = CIoU loss, confidence loss = binary cross-entropy, classification loss = cross-entropy).

[0219] Optimizer: AdamW optimizer, with an initial learning rate of 1e. -4 Every 100 rounds, it decays to 1 / 10 of its original value;

[0220] Training process: Input an expanded sample set, iterate for 300 rounds, and evaluate the accuracy (mAP@0.5) on the validation set every 20 rounds.

[0221] Save the weights of the first detection model with the highest mAP on the validation set (YOLO_best.pt). This first detection model can output the bounding box, confidence score, and category of suspected fireworks in a single frame image.

[0222] The formula for the cross-entropy loss during training of the physics-aware convolutional temporal model is as follows:

[0223]

[0224] This loss function optimizes the model parameters through backpropagation during training to ensure that the classifier can accurately distinguish between fire and non-fire sequences.

[0225] Based on the output of the first detection model, a physical perception convolutional temporal model is trained to improve dynamic feature discrimination ability and suppress static interference. The input during training is a continuous frame sequence (5 frames per segment, corresponding to M=5) from the expanded sample set. The first detection model detects suspected regions in each frame, crops the ROI regions (64×64 size), extracts the average brightness of the ROI in each frame, and generates a brightness feature sequence. ;

[0226] Loss function: Total loss ( For classification cross-entropy loss, For physical regularization, =0.1);

[0227] Optimizer: SGD optimizer, learning rate = 5e -5 The training process was iterated for 200 rounds, and the temporal classification accuracy was evaluated on the validation set every 15 rounds.

[0228] Model saving: Save the weights of the second detection model with the highest accuracy on the validation set.

[0229] The trained YOLO detector is concatenated with the Physics-aware convolutional temporal model (Phy-ConvLSTM module) for joint training to optimize feature fusion performance.

[0230] The joint model architecture takes a sequence of 5 consecutive image frames as input; the first detection model outputs the bounding box and confidence score for each frame. Then, the ROI is cropped to generate a luminance feature sequence, and finally, the Phy-ConvLSTM module outputs the timing correction coefficients. To obtain the final confidence level ;

[0231] When using a joint training configuration, the total loss is: ( The YOLO detection loss of the first detection model is... =0.3, =0.1), This is the physical modulation regularization term for the Phy-ConvLSTM module, which indicates that the "firework frequencies learned by the network" are close to the "physical frequency distribution".

[0232] By jointly optimizing the losses in these two stages, both detection accuracy and robustness of temporal classification can be improved, ultimately achieving end-to-end fireworks recognition. The entire training process employs optimization algorithms such as gradient descent, iteratively updating model parameters on a large dataset to minimize the total loss and ensure the system's generalization ability in real-world scenarios.

[0233] The training strategy involves loading the optimal weights from two separate training phases, reducing the learning rate to 1e. -5 The training process was iterated for 100 rounds, with a focus on optimizing the confidence fusion coefficient and the physical constraint parameters of the Phy-ConvLSTM module.

[0234] The model performance (false positive rate in nighttime scenes, recall rate for fireworks detection, mAP@0.5) was evaluated on the test set, and the best-performing physical perception convolutional temporal model was saved.

[0235] For training INT8 quantization is performed to optimize the computational efficiency of convolutional layers and ensure that the inference speed meets real-time requirements (single-frame inference ≤30ms).

[0236] Mode switching logic: An embedded ambient light intensity judgment module automatically switches to night mode when the average image brightness L < 50, otherwise it switches to day mode;

[0237] In daytime mode, only the first detection model (also known as the YOLO detector) is enabled; in nighttime mode, the first detection model + physical perception convolutional temporal model + frequency domain analysis + edge jitter detection are enabled. If all four conditions are met and the same condition is met for M=5 consecutive frames, it is determined to be a fire event.

[0238] Test set validation: Validate the model performance on the test set, requiring a false positive rate of ≤3% at night, a recall rate of ≥95% for smoke and fire detection, and mAP@0.5≥0.92; collect video streams in real-world scenarios such as roads, construction sites, and tunnels to test the model's stability and detection accuracy during day and night switching, and record and fix a small number of false positive / false negative issues in extreme scenarios.

[0239] Based on the above embodiments, Figure 5The detailed process of the day and night dual-mode smoke detection method in this embodiment of the invention is as follows: a dual-channel camera device (visible light camera and near-infrared supplementary light camera) is used to determine the current acquisition mode and acquire color image sequences or grayscale image sequences. The color image sequence or grayscale image sequence includes single-frame images of consecutive frames, which are then processed separately. In daytime mode, a first detection model is used to determine fire events. In nighttime mode, a combination of the first detection model, a physical perception convolutional temporal model, brightness judgment, and jitter judgment is used to determine fire events. Then, it is decided whether to issue an alarm. If so, an audible and visual alarm can be triggered through the control and alarm module, and the alarm can be reported to the monitoring center to achieve PTZ linkage. The images and data used in this judgment result can be used for online updates of the aforementioned model.

[0240] This invention also proposes a day-night dual-mode fireworks detection device, the principle of which is similar to the day-night dual-mode fireworks detection method, and will not be described in detail here.

[0241] Figure 6 This is a schematic diagram of a day / night dual-mode smoke detection device in an embodiment of the present invention, comprising:

[0242] The acquisition mode determination module 601 is used to determine the acquisition mode based on the ambient light conditions at the current moment.

[0243] The daytime detection module 602 is used to input the color image sequence captured by the visible light camera into the first smoke detection model if the acquisition mode is daytime mode, obtain the first detection result, and determine whether there is a fire event based on the first detection result. The first smoke detection model is a convolutional neural network structure.

[0244] The night detection module 603 is used to input the gray image sequence acquired by the near-infrared supplementary light camera into the first smoke detection model to obtain the second detection result if the acquisition mode is night mode, and input the second detection result into the physical perception convolutional temporal model to obtain the third detection result; if the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected smoke area in each frame of gray image in the third detection result conform to the smoke spectrum pattern and the edge jitter amount meets the smoke dynamic features, it is determined that there is a fire event; the second detection result includes the bounding box and confidence of each frame of gray image, and the physical perception convolutional temporal model is used to generate the temporal correction coefficient of each frame of gray image through gating modulation, and to filter gray images through the temporal correction coefficient to obtain the third detection result.

[0245] In one embodiment, the first detection result includes the bounding box and confidence level of each frame of color image;

[0246] The daytime detection module is used for:

[0247] If the confidence level of a consecutive preset number of color images exceeds a first threshold, and the shape and size of the bounding boxes of the consecutive preset number of color images meet preset requirements, a fire event is determined to exist.

[0248] In one embodiment, the second detection result includes the bounding box and confidence level of each grayscale image frame;

[0249] The physical-aware convolutional temporal model includes:

[0250] The connecting module is used to crop the suspected fireworks area of ​​each gray image based on the bounding box of each gray image and construct a brightness feature sequence of consecutive frames.

[0251] The calibration module is used to generate temporal correction coefficients for each frame of grayscale image based on the brightness feature sequence and through gated modulation.

[0252] The fusion module is used to perform weighted fusion of the confidence score and temporal correction coefficient of each grayscale image to obtain the final confidence score of each grayscale image.

[0253] The filtering module is used to filter the bounding boxes based on the final confidence scores of all frames of grayscale images, and obtain the filtered bounding boxes and their final confidence scores to form the third detection result.

[0254] In one embodiment, the calibration module includes:

[0255] The physical frequency feature extraction module is used to generate a physical consistency score for fireworks based on the brightness feature sequence.

[0256] The gated modulation module is used to obtain a physically constrained basic input gate and a basic forget gate based on convolution operations, and to perform physical constraint modulation on the basic input gate and the basic forget gate using the pyrotechnic physical consistency score to obtain a modulated input gate and a modulated forget gate.

[0257] The cell state update module is used to update the cell state of the current frame based on the modulation input gate, the modulation forget gate, the hidden state of the previous frame, the brightness feature matrix of the current frame, and the cell state of the previous frame.

[0258] The hidden state output module is used to calculate the output gate based on the brightness feature matrix of the current frame and the hidden state of the previous frame, and to calculate the hidden state of the current frame based on the output gate and the cell state of the current frame.

[0259] The mapping module is used to map the hidden state of the current frame to the timing correction coefficient of the current frame.

[0260] In one embodiment, the physical frequency feature extraction module is used for:

[0261] Calculate the brightness feature change matrix based on the brightness feature matrix of the current frame and the brightness feature matrix of the previous frame in the brightness feature sequence.

[0262] Calculate the spectrum of brightness feature changes based on the brightness feature change matrix;

[0263] Calculate the dominant frequency component of the spectrum of brightness characteristic changes;

[0264] The physical consistency score of the fireworks is calculated based on the main frequency component and the preset frequency band range of the fireworks flashing.

[0265] In one embodiment, the night detection module is used to:

[0266] Deep features of the brightness feature sequence of the suspected fireworks area in each frame of the gray image in the second detection result are extracted by a one-dimensional convolutional temporal coding network to obtain the temporal feature vector;

[0267] Calculate the optimal order of the fractional Fourier transform based on the temporal feature vector;

[0268] Based on the optimal order, an adaptive fractional Fourier transform is performed on the brightness feature sequence to obtain the frequency domain signal;

[0269] The adaptive fractional Fourier transform features are extracted from the frequency domain signal. These features include fractional energy, fractional spectral entropy, spectral bandwidth, and main peak value.

[0270] The adaptive fractional Fourier transform features are compared with the corresponding threshold range to determine whether they conform to the fireworks spectrum pattern.

[0271] In one embodiment, the night detection module is used to:

[0272] Extract the edge intensity matrix of the suspected fireworks region from any two consecutive grayscale images, and calculate the edge difference matrix based on the Sobel operator;

[0273] The edge difference matrix is ​​filtered according to the preset minimum difference threshold to obtain the filtered edge difference matrix;

[0274] Perform global statistics on the selected edge difference matrix to obtain the edge jitter amount;

[0275] Based on the preset jitter threshold, determine whether the edge jitter amount meets the dynamic characteristics of fireworks.

[0276] In one embodiment, the device further includes a training module 604, used for:

[0277] Obtain the original real sample set, no-fire night scene images, and smoke template images. The original real sample set includes color image samples and gray image samples of smoke.

[0278] Based on the original real sample set, images of fireless nighttime scenes, and smoke template images, an expanded sample set is obtained through image enhancement and synthesis.

[0279] The first detection model is trained based on the expanded sample set, and the trained first detection model is obtained.

[0280] Based on the trained first detection model, train the physical perception convolutional temporal model to obtain the trained physical perception convolutional temporal model.

[0281] The trained first detection model and the trained physical perception convolutional temporal model are concatenated and then jointly trained.

[0282] In one embodiment, the training module is used to:

[0283] Based on images of fireless nighttime scenes and smoke template images, single-frame images from the original real sample set are enhanced and synthesized to obtain an expanded sample set, including:

[0284] A single-frame image from the original real sample set is enhanced to obtain an enhanced image. The enhancement includes at least one of geometric transformation enhancement, geometric transformation enhancement, and infrared-simulated enhancement.

[0285] After adding dynamic brightness perturbation to the fireworks template image, the image is synthesized with a nighttime scene image without fire to obtain a multi-scene fused image, which is then labeled.

[0286] Based on preset text prompts and image constraints, an open-source diffusion model is used to obtain and label images generated by the diffusion model.

[0287] Representative fireworks images are selected from single-frame images in the original real sample set, scene text labels are generated, and a continuous frame sequence is generated by combining single-frame images and scene text labels. Temporal smoothing correction is performed on the continuous frame sequence to obtain a synthetic fireworks video. The synthetic fireworks video is then split into frames to obtain a composite fireworks video image.

[0288] The original real sample set, enhanced images, multi-scene fusion generated images, diffusion model generated images, and fireworks video composite images are merged and deduplicated to form an expanded sample set.

[0289] In summary, the method and apparatus proposed in this invention include a day / night dual-mode switching mechanism: integrating visible light and near-infrared imaging modes, using color images to detect fires during the day and automatically switching to infrared supplementary imaging at night, ensuring monitoring capabilities under low light conditions while avoiding the use of high-cost thermal imaging equipment. A nighttime smoke and fire recognition method based on brightness fluctuations is also proposed: addressing the problem of lack of color information and easy confusion with other light sources in nighttime grayscale images, this invention introduces time-series analysis of brightness intensity to extract the unique flicker frequency features of smoke and fire to distinguish them from ordinary lights. A two-stage target detection fusion temporal module is implemented: based on the separately trained first detection model, a physical perception convolutional temporal model is added, enabling the physical perception convolutional temporal model to learn the dynamic patterns of smoke flickering and smoke drifting from video frame sequences. This effectively improves the robustness of detection and the sensitivity to small flames and intermittent fires. A two-stage detection system based on the first smoke and fire detection model and the physical perception convolutional temporal model, along with spectral fluctuation analysis results and dynamic smoke and fire determination based on brightness edge jitter, is designed. The alarm is triggered only when smoke and fire pass by simultaneously at night, significantly reducing the false alarm rate. This multi-source fusion strategy comprehensively considers both the static appearance and dynamic characteristics of smoke and fire, making it more reliable than single-feature judgment methods. A fire monitoring image dataset covering day and night scenes was constructed, and synthetic data was innovatively introduced to enhance nighttime smoke and fire samples. Through image overlay and simulation, grayscale images under near-infrared illumination were generated for model training, overcoming the challenge of insufficient real nighttime fire data.

[0290] This invention also provides a computer device. Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 700 includes a memory 710, a processor 720, and a computer program 730 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 730, it implements the above-mentioned day and night dual-mode smoke detection method.

[0291] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described day-night dual-mode smoke detection method.

[0292] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described day and night dual-mode smoke detection method.

[0293] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0294] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0295] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0296] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0297] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A day / night dual-mode smoke detection method, characterized in that, include: Determine the data acquisition mode based on the current ambient lighting conditions; If the acquisition mode is daytime mode, the color image sequence acquired by the visible light camera is input into the first smoke detection model to obtain the first detection result. Based on the first detection result, it is determined whether there is a fire event. The first smoke detection model is a convolutional neural network structure. If the acquisition mode is night mode, the gray image sequence acquired by the near-infrared supplementary light camera is input into the first smoke detection model to obtain the second detection result. The second detection result is then input into the physical perception convolutional temporal model to obtain the third detection result. If the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected fire area in each frame of gray image in the third detection result conform to the fire spectrum pattern and the edge jitter satisfies the fire dynamic features, it is determined that a fire event exists; the second detection result includes the bounding box and confidence score of each frame of gray image, and the physical perception convolutional temporal model is used to generate temporal correction coefficients for each frame of gray image through gating modulation, and gray images are filtered through the temporal correction coefficients to obtain the third detection result; The second detection result includes the bounding box and confidence score of each grayscale image frame; The physical perception convolutional temporal model combines the spatial feature extraction capability of convolutional neural networks with the temporal series modeling advantages of LSTM. The physical perception convolutional temporal model includes: a concatenation module, used to crop the suspected fireworks region of each grayscale image based on the bounding box of each frame and construct a brightness feature sequence of consecutive frames; a calibration module, used to generate temporal correction coefficients for each grayscale image based on the brightness feature sequence through gated modulation; a fusion module, used to perform weighted fusion of the confidence level and temporal correction coefficients of each grayscale image to obtain the final confidence level of each grayscale image; and a filtering module, used to filter the bounding boxes according to the final confidence levels of all grayscale images to obtain the filtered bounding boxes and the final confidence levels, forming a third detection result. The calibration module is used to: calculate the brightness feature change matrix based on the brightness feature matrix of the current frame and the brightness feature matrix of the previous frame in the brightness feature sequence; calculate the brightness feature change spectrum based on the brightness feature change matrix; calculate the dominant frequency component of the brightness feature change spectrum; calculate the physical consistency score of fireworks based on the dominant frequency component and the preset frequency band range of fireworks flashing; and generate the temporal correction coefficient of each frame of gray image through gated modulation using the physical consistency score of fireworks.

2. The method as described in claim 1, characterized in that, The first detection result includes the bounding box and confidence level of each color image frame; Based on the first detection result, determine whether a fire incident exists, including: If the confidence level of a consecutive preset number of color images exceeds a first threshold, and the shape and size of the bounding boxes of the consecutive preset number of color images meet preset requirements, a fire event is determined to exist.

3. The method as described in claim 1, characterized in that, The calibration module includes: The physical frequency feature extraction module is used to generate a physical consistency score for fireworks based on the brightness feature sequence. The gated modulation module is used to obtain a physically constrained basic input gate and a basic forget gate based on convolution operations, and to perform physical constraint modulation on the basic input gate and the basic forget gate using the pyrotechnic physical consistency score to obtain a modulated input gate and a modulated forget gate. The cell state update module is used to update the cell state of the current frame based on the modulation input gate, the modulation forget gate, the hidden state of the previous frame, the brightness feature matrix of the current frame, and the cell state of the previous frame. The hidden state output module is used to calculate the output gate based on the brightness feature matrix of the current frame and the hidden state of the previous frame, and to calculate the hidden state of the current frame based on the output gate and the cell state of the current frame. The mapping module is used to map the hidden state of the current frame to the timing correction coefficient of the current frame.

4. The method as described in claim 1, characterized in that, Also includes: Deep features of the brightness feature sequence of the suspected fireworks area in each frame of the gray image in the second detection result are extracted by a one-dimensional convolutional temporal coding network to obtain the temporal feature vector; Calculate the optimal order of the fractional Fourier transform based on the temporal feature vector; Based on the optimal order, an adaptive fractional Fourier transform is performed on the brightness feature sequence to obtain the frequency domain signal; The adaptive fractional Fourier transform features are extracted from the frequency domain signal. These features include fractional energy, fractional spectral entropy, spectral bandwidth, and main peak value. The adaptive fractional Fourier transform features are compared with the corresponding threshold range to determine whether they conform to the fireworks spectrum pattern.

5. The method as described in claim 1, characterized in that, Also includes: Extract the edge intensity matrix of the suspected fireworks region from any two consecutive grayscale images, and calculate the edge difference matrix based on the Sobel operator; The edge difference matrix is ​​filtered according to the preset minimum difference threshold to obtain the filtered edge difference matrix; Perform global statistics on the selected edge difference matrix to obtain the edge jitter amount; Based on the preset jitter threshold, determine whether the edge jitter amount meets the dynamic characteristics of fireworks.

6. The method as described in claim 1, characterized in that, Also includes: Obtain the original real sample set, no-fire night scene images, and smoke template images. The original real sample set includes color image samples and gray image samples of smoke. Based on the original real sample set, images of fireless nighttime scenes, and smoke template images, an expanded sample set is obtained through image enhancement and synthesis. The first detection model is trained based on the expanded sample set, and the trained first detection model is obtained. Based on the trained first detection model, train the physical perception convolutional temporal model to obtain the trained physical perception convolutional temporal model. The trained first detection model and the trained physical perception convolutional temporal model are concatenated and then jointly trained.

7. The method as described in claim 6, characterized in that, Based on images of fireless nighttime scenes and smoke template images, single-frame images from the original real sample set are enhanced and synthesized to obtain an expanded sample set, including: A single-frame image from the original real sample set is enhanced to obtain an enhanced image. The enhancement includes at least one of geometric transformation enhancement, geometric transformation enhancement, and infrared-simulated enhancement. After adding dynamic brightness perturbation to the fireworks template image, the image is synthesized with a nighttime scene image without fire to obtain a multi-scene fused image, which is then labeled. Based on preset text prompts and image constraints, an open-source diffusion model is used to obtain and label images generated by the diffusion model. Representative fireworks images are selected from single-frame images in the original real sample set, scene text labels are generated, and a continuous frame sequence is generated by combining single-frame images and scene text labels. Temporal smoothing correction is performed on the continuous frame sequence to obtain a synthetic fireworks video. The synthetic fireworks video is then split into frames to obtain a composite fireworks video image. The original real sample set, enhanced images, multi-scene fusion generated images, diffusion model generated images, and fireworks video composite images are merged and deduplicated to form an expanded sample set.

8. A day / night dual-mode smoke detection device, characterized in that, include: The acquisition mode determination module is used to determine the acquisition mode based on the current ambient lighting conditions. The daytime detection module is used to input the color image sequence captured by the visible light camera into the first smoke detection model if the acquisition mode is daytime mode, obtain the first detection result, and determine whether there is a fire event based on the first detection result. The first smoke detection model is a convolutional neural network structure. The night detection module is used to input the gray image sequence captured by the near-infrared supplementary light camera into the first smoke detection model if the acquisition mode is night mode, to obtain the second detection result, and input the second detection result into the physical perception convolutional temporal model to obtain the third detection result; If the adaptive fractional Fourier transform features of the brightness feature sequence of the suspected fire area in each frame of gray image in the third detection result conform to the fire spectrum pattern and the edge jitter satisfies the fire dynamic features, it is determined that a fire event exists; the second detection result includes the bounding box and confidence score of each frame of gray image, and the physical perception convolutional temporal model is used to generate temporal correction coefficients for each frame of gray image through gating modulation, and gray images are filtered through the temporal correction coefficients to obtain the third detection result; The second detection result includes the bounding box and confidence score of each grayscale image frame; The physical perception convolutional temporal model combines the spatial feature extraction capability of convolutional neural networks with the temporal series modeling advantages of LSTM. The physical perception convolutional temporal model includes: a concatenation module, used to crop the suspected fireworks region of each grayscale image based on the bounding box of each frame and construct a brightness feature sequence of consecutive frames; a calibration module, used to generate temporal correction coefficients for each grayscale image based on the brightness feature sequence through gated modulation; a fusion module, used to perform weighted fusion of the confidence level and temporal correction coefficients of each grayscale image to obtain the final confidence level of each grayscale image; and a filtering module, used to filter the bounding boxes according to the final confidence levels of all grayscale images to obtain the filtered bounding boxes and the final confidence levels, forming a third detection result. The calibration module is used to: calculate the brightness feature change matrix based on the brightness feature matrix of the current frame and the brightness feature matrix of the previous frame in the brightness feature sequence; calculate the brightness feature change spectrum based on the brightness feature change matrix; calculate the dominant frequency component of the brightness feature change spectrum; calculate the physical consistency score of fireworks based on the dominant frequency component and the preset frequency band range of fireworks flashing; and generate the temporal correction coefficient of each frame of gray image through gated modulation using the physical consistency score of fireworks.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images