Flame detection method and apparatus suitable for low quality dual spectrum images

Abstract

The present application belongs to the field of fire warning, and particularly relates to a flame detection method suitable for low-quality dual-spectrum images and a device thereof. The method introduces a frequency domain complementary fusion mechanism FCFM and a soft perception invalid mode gating mechanism SPMG in the backbone network of a visible light and infrared dual-branch flame detection model. The former is used for differential filtering enhancement processing of visible light features and infrared features in the frequency domain, thereby significantly improving the learning ability of the model to the effective information in the two types of data. The latter adaptively fuses the features of the two modes according to the global average intensity of different modal data, and replaces the invalid mode through the projection of the effective mode; thereby avoiding the influence of invalid mode data on the model performance in the training and inference stages. Applying one or all of FCFM and SPMG to the dual-branch flame detection model helps to improve the prediction accuracy of the model under low-quality samples and overcome the defects of insufficient robustness of traditional schemes.

Description

Technical Field

[0001] This invention belongs to the field of fire early warning, specifically relating to a flame detection method suitable for low-quality dual-spectral images, and a corresponding flame recognition device. Background Technology

[0002] Traditional fire detection and identification typically utilize temperature and smoke sensors. However, these methods suffer from limitations in real-time performance and are prone to false positives, leading to losses of manpower and resources. To address this issue, a series of computer vision-based methods have been proposed and applied to fire detection. Among them, the YOLO series is widely used due to its lightweight design and fast response speed. While computer vision methods offer good performance, they still sometimes fail to achieve ideal detection results in practical applications. This is due to factors such as the scarcity of fire datasets and poor image quality.

[0003] To address the limitations of fire detection technology due to the quantity and quality of datasets, researchers have proposed numerous improved methods and models for flame detection. However, these solutions have not effectively eliminated interference sources in visible light flame image detection. In image detection, particularly pedestrian detection, images at night and in low light often lack salient features, making it difficult for models to identify pedestrians. Infrared images, on the other hand, can capture salient features at night. Therefore, some researchers have combined visible light and infrared images for detection to improve model accuracy and fusion detection performance in low-light conditions. With technological advancements, fusion detection has been broadly categorized into three types: early-stage fusion, mid-stage fusion, and late-stage fusion. Mid- and late-stage fusion shows the most significant results. However, mid- and late-stage fusion typically employs dual-branch or even multi-branch backbone networks, leading to insufficient timeliness of the network model. Early-stage fusion aims to fuse visible light and infrared images before the image is input into the backbone network. This fusion method is not conducive to the interaction of visible light and infrared image features, resulting in inferior performance compared to mid- and late-stage methods. Inspired by pedestrian image fusion detection, a number of flame detection technologies have emerged that can fuse visible light and infrared images. Because flames are highly salient in infrared images and can eliminate certain interference sources, they have attracted widespread attention. Currently, the commonly used method is to perform feature-level fusion, i.e., mid-term feature fusion, using CNN networks and similar techniques. While there are abundant results in pedestrian or vehicle detection based on bispectral images, transferring this work to bispectral flame detection has not achieved the expected results. This is because in fields such as pedestrian detection, there is a tendency to enhance features first and then fuse them for detection, but in bispectral flame detection, there is no method for enhancing flame features specific to each modality.

[0004] In the field of image processing, the Fourier transform transforms features from the spatial domain to the frequency domain. In the frequency domain, "frequency" does not represent temporal transformation but rather the degree of spatial variation in grayscale values ​​(pixel brightness). Therefore, regions with slow and flat grayscale changes are low-frequency components, while regions with rapid and drastic grayscale changes are high-frequency components. Traditional CNNs also incorporate frequency domain features to address the issues of feature redundancy and insufficient receptive field in the spatial domain. In object detection, many researchers have introduced attention mechanisms into the frequency domain, allowing the machine to autonomously retain or learn more important frequency domain features, or set threshold mechanisms to retain or remove redundant features. In multimodal detection, some researchers fuse information from two modalities in the frequency domain or remove redundant modal information to achieve better feature fusion results. However, in existing schemes, the Fourier transform mainly focuses on feature selection and enhancement. But for multimodal images, due to modal differences, the distribution and differences of frequency domain features after Fourier transform for different modalities are quite significant. For example, targets such as pedestrians or vehicles, due to their relatively low heat content, often exhibit similar modal characteristics in both infrared and visible light modes, resulting in a uniform modal correlation distribution. However, flames, with their larger heat source, show significant differences in modal characteristics between infrared and visible light modes, and the correlation between the two modes is markedly different due to the unique characteristics of flames, such as their high brightness and heat. Therefore, traditional frequency domain enhancement techniques are less effective in flame detection. Summary of the Invention

[0005] To address the issue that traditional image enhancement and feature fusion mechanisms in target detection are not suitable for flame detection, which leads to insufficient detection accuracy and robustness of dual-modal flame detection models under low-quality input, this invention provides a flame detection method and device suitable for low-quality dual-spectral images.

[0006] This invention is achieved using the following technical solution:

[0007] A flame detection method suitable for low-quality dual-spectral images includes the following steps:

[0008] I. A flame detection model is constructed, comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. The hierarchical fusion module includes a frequency-domain complementary fusion mechanism (FCFM) introduced in a designated layer of the dual-branch backbone network. The dual-branch backbone network is used to extract visible light and infrared features at different scales. FCFM performs high-pass filtering on visible light features in the frequency domain using an adjustable-threshold high-pass filter, and low-pass filtering on infrared features in the frequency domain using an adjustable-threshold low-pass filter. Then, the filtered features are residually concatenated with the original features, and then subjected to 1×1 convolution, batch normalization, and SiLU activation to obtain the enhanced features of the corresponding branches. The enhanced features of the two branches are concatenated to form the hierarchical fusion features at the corresponding scale. The neck network is used to perform multi-scale fusion of the input features from various levels; the prediction head is used to output the prediction result based on the output features of the neck network.

[0009] 2. Flame tags are added to the synchronously acquired visible light and infrared images to form samples; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image. The flame detection model is trained using the sample dataset, and the trained flame detection model is then used to detect flames on the input samples.

[0010] In another embodiment provided by the present invention, the flame detection method applicable to low-quality dual-spectral images includes the following steps:

[0011] I. Constructing a flame detection model comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. The hierarchical fusion module includes a soft-sensing ineffective mode gating mechanism (SPMG) introduced in a specified layer of the dual-branch backbone network. The dual-branch backbone network is used to extract visible light and infrared features at different scales. SPMG is used to first calculate the global average intensity of the visible light and infrared features separately, and generate an intensity-adaptive mask for each branch based on a preset intensity threshold; then, the visible light and infrared features are projected onto the other branch to obtain the corresponding compensated features. and The bisexual feature is obtained by stitching together the visible light and infrared features. F dual Then, the bimodal effective weights are synthesized based on the masks of the two branches. Visible light effective weight only and infrared effective weights only Finally, the hierarchical fusion features output by the soft-perception invalid mode gating mechanism are generated. F out : The neck network is used to perform multi-scale fusion of the input features from various levels; the prediction head is used to output the prediction result based on the output features of the neck network.

[0012] 2. Flame tags are added to the synchronously acquired visible light and infrared images to form samples; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image. The flame detection model is trained using the sample dataset, and the trained flame detection model is then used to detect flames on the input samples.

[0013] In the third embodiment provided by this invention, the flame detection method applicable to low-quality dual-spectral images includes the following steps:

[0014] I. Constructing a flame detection model comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. The hierarchical fusion module includes a frequency-domain complementary fusion mechanism (FCFM) and a soft-sensing ineffective mode gating mechanism (SPMG) introduced in designated layers of the dual-branch backbone network. The dual-branch backbone network is used to extract visible light and infrared features at different scales. FCFM performs high-pass filtering on visible light features in the frequency domain using an adjustable-threshold high-pass filter, and low-pass filtering on infrared features in the frequency domain using an adjustable-threshold low-pass filter; then, the filtered features are residually concatenated with the original features, and after 1×1 convolution, batch normalization, and SiLU activation, the enhanced features of the corresponding branches are obtained; the enhanced features of the two branches are concatenated to form the bi-functional features at the corresponding scale. F dual .

[0015] SPMG is used to first calculate the global average intensity of the enhanced visible light and infrared features separately, and then generate an intensity-adaptive mask for each branch based on a preset intensity threshold; finally, the enhanced visible light and infrared features are projected onto the other branch to obtain the corresponding compensation features. and The effective weights for the two modes are synthesized based on the masks of the two branches. Visible light effective weight only and infrared effective weights only This leads to the hierarchical fusion features output by the soft-perception invalid mode gating mechanism. F out : ;

[0016] The neck network is used to perform multi-scale fusion of the input features from various levels; the prediction head is used to output the prediction result based on the output features of the neck network.

[0017] 2. Flame tags are added to the synchronously acquired visible light and infrared images to form samples; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image. The flame detection model is trained using the dataset, and the trained flame detection model is used to detect flames on the input samples.

[0018] The present invention also includes a flame recognition device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a flame detection model trained as described in the aforementioned flame detection method for low-quality dual-spectral images. The flame detection model is used to generate detection results of flames contained in the input visible light image and infrared image.

[0019] The technical solution provided by this invention has the following beneficial effects:

[0020] To address the impact of low-quality images on the detection accuracy of dual-branch flame detection networks, this invention proposes a novel frequency-domain complementary fusion mechanism (FCFM) and a soft-sensing invalid mode gating mechanism (SPMG). This enables frequency-domain enhancement and adaptive fusion of features. This invention significantly improves the performance of dual-modal flame detection models in both singular and collaborative applications of FCFM and SPMG.

[0021] The FCFM mechanism of this invention first utilizes the sensitivity of Fourier transform to grayscale value transformation to perform time-frequency conversion on features. Then, it suppresses high-frequency features of visible light images and enhances their low-frequency features in the frequency domain. This further enhances high-frequency features of visible light image texture and shape details, weakens low-frequency smooth area features, and increases the shape and texture differences between flame targets and other targets. For infrared images, it enhances low-frequency features and suppresses high-frequency features, allowing the model to focus more on judging the overall region rather than the image edge shape when learning infrared flame features. Finally, the enhanced visible light and infrared features are inversely transformed and fused to obtain enhanced fused bi-branch features.

[0022] The SPMG mechanism calculates the global average intensity of the input features and statistically analyzes the energy density of the feature map. This invention designs a dedicated control function based on the intensity index, with specific hyperparameters adjusting the hardness or softness of the gating. This allows for parameter adjustment during inference, enabling the use of a hard gating mechanism to allow another modality to take over and achieve feature fusion when one modality fails. The SPMG mechanism reduces the impact of feature dilution caused by introducing failed modes, enhancing the robustness of dual-spectral flame detection in extreme environments. Simultaneously, the soft gating method ensures smooth end-to-end training. Attached Figure Description

[0023] Figure 1 This is an architecture diagram of a traditional dual-modal flame detection model.

[0024] Figure 2 This is an architecture diagram of the flame detection model with frequency domain complementary fusion mechanism FCFM provided in Embodiment 1 of the present invention.

[0025] Figure 3 This is a schematic diagram of the frequency domain complementary fusion mechanism proposed in Embodiment 1 of the present invention.

[0026] Figure 4 This is a comparison chart of infrared and visible light images under different exposure levels.

[0027] Figure 5 This is a visualization of a bispectral image after high-frequency and low-frequency filtering.

[0028] Figure 6 This is an architecture diagram of the flame detection model that introduces the soft-sensing invalid modal gating mechanism SPMG provided in Embodiment 2 of the present invention.

[0029] Figure 7 This is an architecture diagram of the flame detection model that simultaneously incorporates FCFM and SPMG, as provided in Embodiment 3 of the present invention.

[0030] Figure 8 This is a heatmap of the enhanced feature images obtained using FCFM in different scenarios during the simulation experiment.

[0031] Figure 9 This is a heatmap of the feature images of different schemes after fusion at the first layer in the simulation experiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] In their research, the technicians of this embodiment discovered that:

[0034] (i) Visible and near-infrared image data of flames differ significantly from bimodal data in traditional target detection tasks such as pedestrians and vehicles. For example, in visible light images, flames are bright and irregularly shaped targets. This characteristic weakens the target features of people and vehicles in darker environments; conversely, flames are more prominent in darker environments. Furthermore, the brightness of flames causes nearby objects in the image to lose edge texture features, and they are more easily confused with objects such as LED lights and traffic lights. In well-lit conditions, targets such as people and vehicles are particularly noticeable, while small flames become assimilated into the surrounding bright environment; for example, they are interfered with by specular reflections and white walls. In the infrared modality, although pedestrians and vehicles have a certain temperature, it is not significantly different from the surrounding environment. Therefore, pedestrians do not stand out in infrared images, while flames are particularly prominent in infrared images due to their high heat and concentrated heat distribution, and their shape differs somewhat from that in visible light. Its high brightness and high heat characteristics result in flames having an internal grayscale value of 255, i.e., a white spot region, in addition to the obvious edge shape information. Therefore, the visible light infrared data of flames differs significantly from the infrared data of flames from pedestrians and vehicles. When fusing the two-modal features of flames, adaptive modal processing and fusion should be performed according to the characteristics of flames to make them more suitable for multimodal detection.

[0035] (II) Current methods for enhancing features mainly focus on attention and frequency domain spatial feature selection. Attention methods, which suppress background noise and focus on key information, are essentially adaptive feature weighting and fusion processes. For example, channel attention and spatial attention determine strong correlations through backpropagation to automatically learn the importance weights of each channel and the weight distribution in two-dimensional space. However, in lightweight target detection, such as flame detection, this complex attention mechanism is often no longer applicable. For algorithms that need to be deployed on edge devices, even with a more lightweight attention mechanism, stacking attention can still slow down the model or even result in negligible accuracy improvement. Furthermore, for tiny flames in flame target detection, channel attention and global average pooling compress spatial information, leading to diluted target features and missed detections.

[0036] Enhancement methods for frequency domain features involve transforming spatial domain features into frequency domain features, performing feature filtering on the frequency domain features, and finally inversely transforming them back into spatial domain features. Transformation methods include Fourier transform and discrete wavelet transform. Compared to attention methods, this enhancement approach incurs less computational cost. However, while frequency domain feature enhancement methods can effectively extract complementary features, they also suffer from high noise sensitivity.

[0037] To address this issue, the technicians in this embodiment utilize the characteristic of Fourier transform being sensitive to grayscale values ​​when allocating high and low frequency features. They apply it to differentiate and enhance the dual-modal features of flames, thereby achieving better feature fusion and improving detection performance.

[0038] (III) In the field of bimodal flame target detection, feature enhancement methods are often designed for specific datasets. However, even models designed for specific datasets and with high scene complexity still experience a large number of false negatives and missed detections when deployed to edge devices. In practical applications, to make the model more generalizable to different scenarios, engineers often re-collect a portion of the dataset for new usage scenarios and retrain the original pre-trained model to adapt to the new environment. However, newly captured data may still cause feature enhancement methods to fail or perform poorly due to factors such as the performance of the capturing equipment and the shooting angle.

[0039] For devices used to capture and detect dual-spectral flame data, parameters such as exposure time and aperture are typically set relatively low to obtain high-quality flame data. This results in darker images, a phenomenon particularly pronounced in model training for indoor scenes. Furthermore, in low-light environments lacking bright or hot targets, one of the visible or near-infrared modes may become ineffective, resulting in a completely black image. For dual-spectral flame detection devices, the infrared mode is more prone to producing completely black images. In this case, only visible light provides effective features, rendering the infrared mode ineffective. If the application scenario cannot avoid having too many ineffective modes, they will dilute the intensity of the other modality during training, causing activation shifts and ultimately leading to numerous false negatives and missed detections.

[0040] To address this situation, this embodiment designs a unique gating mechanism that can effectively identify and handle one of the invalid modes, thereby significantly improving the robustness of the network.

[0041] The architecture of a classic dual-modal flame detection model is as follows: Figure 1 As shown, the system comprises a backbone network, a neck network, and a detection head. The backbone network includes an infrared branch and a visible light branch; these branches are used to extract features at different scales from the input visible light and infrared images, respectively. The backbone network fuses the feature maps extracted from the two branches at each layer and outputs them to the neck network. The neck network further performs multi-scale fusion on the fused features of the input at various sizes and outputs them to the prediction head. The prediction head outputs the prediction result based on the output features of the neck network.

[0042] To address the issue of significant accuracy degradation in dual-modal flame detection models based on visible light and infrared images when the sample image quality for any modality is poor, this embodiment presents a novel mid-term feature enhancement fusion method. This method introduces a special inter-layer fusion module after the visible light and infrared branches of the flame detection network's backbone. This module can be used to perform frequency domain enhancement and complementary fusion of features from the two branches (referred to as Frequency Domain Complementary Fusion Mechanism, FCFM in this embodiment), or to sense invalid modalities and perform adaptive gated fusion (referred to as Soft Sensing Invalid Modality Gating Mechanism, SPMG in this embodiment). This eliminates the impact of low-quality images from any modal input on the model's final detection output.

[0043] In this embodiment, both the FCFM and SPMG mechanisms achieve good technical results when applied independently or in combination to the dual-modal flame detection model. Therefore, the following sections will provide a detailed explanation of the scheme in this embodiment using examples of independent and combined application of the two mechanisms.

[0044] Example 1

[0045] This embodiment provides a flame detection method suitable for low-quality dual-spectral images, which includes the following process:

[0046] I. Construct a flame detection model comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. The hierarchical fusion module includes a frequency-domain complementary fusion mechanism (FCFM) introduced in a specified layer of the dual-branch backbone network.

[0047] The dual-branch backbone network is used to extract visible light and infrared features at different scales. FCFM is used to enhance the features of the two branches in the frequency domain using differentiated strategies, and then concatenate them into corresponding hierarchical fusion features. The neck network is used to perform multi-scale fusion of the input hierarchical fusion features; the prediction head is used to output the prediction result based on the output features of the neck network.

[0048] In practical applications, the backbone network can employ a feature extraction network commonly found in YOLO-based flame detection models. For example, in this embodiment, the YOLOv11n end-to-end detection network is extended to a two-modal backbone network to extract features at different scales. The input two-modal images are first adjusted to 640. 640 resolution; without considering scaling factors, the four downsampling feature extraction layers P1, P2, P3, and P4 of the two-modal backbone network output 160 resolution respectively. 160, 80 80, 40 40, 20 A 20-scale feature map is used, and a hierarchical fusion module is applied to layers P2, P3, and P4 to enhance and fuse the features extracted from the last three layers of the two modalities. This dual-branch backbone network includes a visible light branch and an infrared branch, which are used to process visible light and infrared images of the same size and acquired simultaneously, respectively. In the corresponding layers of the visible light and infrared branches, feature maps of the same size visible light and infrared features can be output. The number of layers in the backbone network may vary in different schemes. To balance lightweight design and accuracy, such as... Figure 2 As shown, FCFM is typically placed in the later layers of the backbone network. For example, in a backbone network with four feature extraction layers, FCFM can be placed in the last three feature extraction layers. The neck network and prediction head part can also be designed directly using schemes from existing flame detection models.

[0049] like Figure 3 As shown, the FCFM designed in this embodiment performs high-pass filtering on visible light features in the frequency domain using an adjustable-threshold high-pass filter, and low-pass filtering on infrared features in the frequency domain using an adjustable-threshold low-pass filter. Then, the filtered features are residually concatenated with the original features, and then subjected to 1×1 convolution, batch normalization, and SiLU activation (implemented via a CBN module) to obtain the enhanced features of the corresponding branches. Finally, the enhanced features of the two branches are concatenated (Cat) to form the hierarchical fusion features at the corresponding scale. .

[0050] To make the improved flame detection model in this embodiment clearer, the working principle of the introduced FCFM is explained below:

[0051] In this embodiment, the FCFM data processing procedure includes two parts: feature enhancement and feature fusion. The feature enhancement is implemented in the frequency domain. To achieve optimal feature enhancement results in both modal images, such as... Figure 4 As shown, technicians placed pedestrians, vehicles, and flames in the same scene and captured bimodal images at different exposure levels. In the captured visible light images, the high brightness of the flames at certain exposure levels weakened the shape and texture features of the flames themselves, as well as the edges of surrounding objects, making them easily confused with similarly colored objects such as car headlights and roadblocks, leading to false detections. In the captured infrared images, pedestrians and vehicles did not stand out in the overall scene. The high heat source of the flames resulted in an overall grayscale value of almost 255 for the flame target, creating a strong contrast with the surrounding low grayscale values, making the target particularly noticeable. In particular, at certain exposure levels, the features of pedestrians and other objects almost disappeared, while only the flames remained in the image. Furthermore, at medium to long distances, the shape of the flames weakened, turning into white spots.

[0052] When enhancing visible light and infrared features in the frequency domain, a Fourier transform of the original features is required. Technicians have discovered that the Fourier transform is extremely sensitive to grayscale values ​​in image features, making it well-suited to the characteristics of flames in both modes. For example... Figure 5 As shown, after visualizing the high-frequency features extracted from visible light and infrared images, technicians discovered that in visible light images, because high-frequency information corresponds to areas of dramatic grayscale changes, the flame shape and texture can be extracted quite completely after high-frequency feature extraction visualization. However, for infrared images, although the flame shape and texture are extracted, there is a significant difference compared to visible light. This difference leads to poor results when fusing high-frequency features from both modalities, as they lack complementarity. Further visualization of low-frequency information extraction reveals that while visible light images retain more information, there is too much redundant information after fusion. In contrast, infrared images have undergone significant feature filtering, resulting in severe degradation of features such as pedestrians and other objects. Conversely, the flame region features are preserved relatively completely. Therefore, for flame datasets, infrared images are more suitable for extracting low-frequency information, as their low-frequency features complement those of visible light, leading to more accurate flame localization after fusion.

[0053] Therefore, in FCFM, this embodiment first designs two frequency domain filters to filter visible light and infrared features respectively. These frequency domain filters filter the time-domain features of visible light and infrared features channel by channel. f ( x, y Perform a two-dimensional discrete Fourier transform (2D-DFT) to obtain the corresponding frequency domain features. F ( u,v The data is then filtered and finally transformed back to the time domain using the inverse Fourier transform (IDFT). The expression for the two-dimensional discrete Fourier transform is as follows:

[0054] ;

[0055] In the above formula, ( x,y () represents the time-domain coordinates of the feature map; u,v () represents the frequency domain coordinates of the feature map; H and W represent the height and width of the feature map; e Represent natural numbers; i It represents the imaginary unit.

[0056] By centering the frequency domain features and calculating the spectral amplitude to determine high and low frequencies, an important piece of information can be obtained: low-frequency features in an image represent slowly changing components, that is, regions where gray-level values ​​change relatively slowly in the gray-level space; these regions mainly represent the overall outline, background, etc. Conversely, high-frequency features represent regions where gray-level values ​​change drastically, mainly manifested in object edges, detailed textures, etc.

[0057] Therefore, based on the image characteristics of the two types of images described above, this embodiment applies a high-pass filter to the visible light features and a low-pass filter to the infrared features. In practical applications, the expression for the high-pass filter is:

[0058] ;

[0059] In the above formula, This represents the visible light characteristics after high-pass filtering; Indicates the backbone network number i Visible light characteristics of the layers; when the backbone network has 4 layers. i =2, 3, 4. Indicates the first i The filter radius of the layer; in a high-pass filter, it is less than The features will be filtered. and These represent the two-dimensional discrete Fourier transform and its inverse transform, respectively; This represents the mask generated by the high-pass filter; The '·' operator represents element-wise multiplication; Real(·) represents the real part extraction operation.

[0060] Accordingly, the expression for the low-pass filter is:

[0061] ;

[0062] In the above formula, This indicates the infrared characteristics after low-pass filtering. Indicates the backbone network number i Infrared characteristics of the layer; Indicates the first i The filter radius of the layer; in a low-pass filter, it is greater than The features will be filtered. This represents the mask generated by the low-pass filter.

[0063] In this embodiment, and Here are a set of hyperparameters to be optimized in the frequency domain complementary fusion mechanism FCFM; their feasible regions are as follows:

[0064] ;

[0065] In the above formula, H i and W i These represent the backbone network's... i The height and width of the visible light or infrared characteristics of the layer.

[0066] In summary, this embodiment, when the hierarchical fusion module only includes FCFM, outputs hierarchical fusion features. The expression is:

[0067] ;

[0068] In the above formula, Concat represents the feature concatenation operation; BN represents the batch normalization operation; Conv 1×1 Represents a 1×1 convolution; Indicates the SiLU activation function; and They represent the characteristics of visible light, respectively. and infrared features Enhanced features; and These represent the visible light characteristics after high-pass filtering in the frequency domain and the infrared characteristics after low-pass filtering in the frequency domain, respectively.

[0069] 2. After adding flame tags to the synchronously acquired visible light and infrared images, samples are formed; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image.

[0070] In this embodiment, considering that the improved flame detection model is specifically designed for scenarios where one of the dual-modal inputs is a low-quality image, it is necessary to collect a corresponding dataset for targeted training. In the practical application of this embodiment, sample images from some publicly available datasets can be obtained, supplemented by samples composed of self-taken images. In the self-taken sample data containing visible light and near-infrared flame images, camera parameters can be adjusted to include some invalid modal infrared or visible light images, i.e., pure black images.

[0071] Third, the flame detection model is trained using the sample dataset, and the trained flame detection model is used to detect flames on input samples where one of the visible light images or infrared images is a low-quality image.

[0072] During the training phase of the flame detection model, the following methods can be used: Figure 1The training strategy of the classic model shown can also be adapted to a two-stage training strategy for newly collected sample data. For example, in the first stage, training can be conducted using a dataset with a low proportion of low-quality samples, and in the second stage, training can be conducted using data with a high proportion of low-quality samples or data consisting entirely of low-quality samples. Furthermore, the model training phase can employ a joint loss function that includes localization loss, confidence loss, and classification loss.

[0073] Example 2

[0074] Based on the scheme in Example 1, this embodiment further provides a new flame detection method suitable for low-quality dual-spectral images. The main difference between this method and the method in Example 1 is that the feature fusion mechanism introduced by the fusion module in the flame detection model is different.

[0075] Specifically, the flame detection method for low-quality dual-spectral images provided in this embodiment has the following steps:

[0076] I. Construct a flame detection model comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. For example... Figure 6 As shown, the hierarchical fusion module includes a soft-perception invalid mode gating mechanism (SPMG) introduced in a designated layer of the dual-branch backbone network. The dual-branch backbone network extracts visible light and infrared features at different scales. SPMG adaptively fuses these features based on the global average intensity of the input features from each branch, reducing the influence of feature components from branches with lower input image quality, thus obtaining the hierarchical fusion features for the corresponding layer. The neck network performs multi-scale fusion of the input hierarchical fusion features; the prediction head outputs the prediction result based on the output features of the neck network.

[0077] Specifically, the SPMG provided in this embodiment is used to first calculate the global average intensity of visible light features and infrared features respectively, and then generate an intensity-adaptive mask for each of the two branches based on a preset intensity threshold.

[0078] In SPMG, pure black images in the dataset are considered invalid modalities, and the invalid modality is determined by whether the input feature map is a pure black image. To determine whether the input feature map is a pure black image, this embodiment chooses to calculate the energy density of the statistical feature map by calculating the global average intensity index of the input feature map, thereby determining the average activation intensity of the current modality.

[0079] Among them, arbitrary mode m Feature map F m ( c , h , w global average intensity Im The calculation formula is as follows:

[0080] ;

[0081] In the above formula, C, H, and W represent the number of channels, height, and width of the feature map, respectively; m=vis and m=nir The times represent the visible light mode and the infrared mode, respectively.

[0082] Based on this, this embodiment designs an adaptive mask for feature images with different global average intensities, thereby ensuring that features with higher global average intensities retain a higher proportion of their feature components, while features with lower global average intensities are more likely to have their feature information masked. Specifically, this embodiment is based on... I m Generate a mask for the corresponding mode. M m The expression is as follows:

[0083]

[0084] In the above formula, This represents the Sigmoid activation function; This indicates the preset intensity threshold. A smaller setting means that the gating mechanism will only be triggered when the image is close to pure black; that is, the global average intensity trigger threshold for a certain modality feature is lowered. In this embodiment... The default value is 0.001. k This indicates the preset soft-gating slope; the larger the slope, the closer it is to a hard switch. In this case, the feature information of the invalid mode will be replaced by the projected feature information of the other mode. k It can control the sensitivity of the gating function to a threshold, that is: control M m A speed close to 1 or 0.

[0085] The visible light and infrared features are then projected onto another branch to obtain the corresponding compensated features. and The bisexual feature is obtained by stitching together the visible light and infrared features. F dual .

[0086] In this embodiment, the purpose of projecting visible light and infrared features is to promptly substitute features from the other branch when the input image of one branch is an invalid mode, thus preventing interference information from the invalid mode features from being introduced into the model and affecting the accuracy of the final prediction result. Specifically, in practical applications, the projection generation formulas for the compensation features of visible light and infrared features are as follows:

[0087] ;

[0088] In the above formula, P represents the projection layer; Represents the visible light characteristic F vis Compensation features projected onto infrared modes; Indicates infrared feature F nir Compensation features projected onto the visible light mode.

[0089] Accordingly, the androgynous feature in this embodiment F dual The calculation formula is as follows:

[0090] F dual = Concat [ F vis ,F nir ];

[0091] Next, this embodiment synthesizes the dual-modal effective weights based on the masks of the two branches. Visible light effective weight only and infrared effective weights only The generation formula is as follows:

[0092] ;

[0093] In the above formula, M vis and M nir These represent visible light mask and infrared mask, respectively.

[0094] Observing the generation formulas for the three types of weights, we can see that the lower the masking rate of any branch, the greater the effective weight of the modality corresponding to that branch; conversely, the higher the masking rate of the input image of any branch, the smaller the effective weight of the modality corresponding to that branch. This design can effectively reduce the influence of invalid modalities and increase the weight of high-quality image features.

[0095] Finally, by combining the three different types of features and their weights, a hierarchical fusion feature is generated from the output of the soft-perception invalid modality gating mechanism. F out :

[0096] .

[0097] 2. After adding flame tags to the synchronously acquired visible light and infrared images, samples are formed; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image.

[0098] This step is similar to the scheme in Example 1, and will not be described again in this embodiment.

[0099] Third, the flame detection model is trained using the sample dataset, and the trained flame detection model is used to detect flames on input samples where one of the visible light images or infrared images is a low-quality image.

[0100] This step is similar to the scheme in Example 1, and will not be described again in this embodiment.

[0101] Example 3

[0102] Based on the schemes in Embodiments 1 and 2, this embodiment further provides a new flame detection method suitable for low-quality dual-spectral images. The difference between this method and the scheme in Embodiment 1 is that it simultaneously introduces the aforementioned frequency domain complementary fusion mechanism FCFM and soft-sensing invalid mode gating mechanism SPMG into the hierarchical fusion module. Specifically, the method includes the following steps:

[0103] I. Construct a flame detection model comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. For example... Figure 7 As shown, the hierarchical fusion module includes a frequency-domain complementary fusion mechanism (FCFM) and a soft-sensing ineffective mode gating mechanism (SPMG) introduced in a designated layer of the dual-branch backbone network. The dual-branch backbone network is used to extract visible light and infrared features at different scales. The hierarchical fusion model first uses FCFM to perform differentiated frequency-domain enhancement on the features extracted from the two branches for different modalities, and then uses SPMG to adaptively fuse the enhanced features according to the global average intensity; thus obtaining hierarchical fused features. The neck network is used to perform multi-scale fusion of the input hierarchical fused features; the prediction head is used to output the prediction result based on the output features of the neck network.

[0104] 2. After adding flame tags to the synchronously acquired visible light and infrared images, samples are formed; the constructed sample dataset contains low-quality samples where one of the visible light or infrared images is a low-quality image.

[0105] This step is similar to the scheme in Example 1, and will not be described again in this embodiment.

[0106] Third, the flame detection model is trained using the dataset, and the trained flame detection model is used to detect flames on input samples where one of the visible light image or infrared image is a low-quality image.

[0107] This step is similar to the scheme in Example 1, and will not be described again in this embodiment.

[0108] In this embodiment, which simultaneously incorporates FCFM and SPMG, the processing strategy for the visible light and infrared features extracted from each layer of the backbone network by the hierarchical fusion module differs somewhat from the data processing strategy when using FCFM or SPMG alone. The following provides a detailed explanation of the data processing strategy of the hierarchical fusion model in this embodiment:

[0109] First, FCFM performs high-pass filtering on visible light features in the frequency domain using an adjustable-threshold high-pass filter, and low-pass filtering on infrared features in the frequency domain using an adjustable-threshold low-pass filter. Then, the filtered features are residually concatenated with the original features, and then subjected to 1×1 convolution, batch normalization, and SiLU activation to obtain enhanced features for the corresponding branches. The enhanced features from the two branches are then concatenated to form bisexual features at the corresponding scale. F dual .Right now:

[0110] .

[0111] SPMG is used to first calculate the global average intensity of the enhanced visible light and infrared features separately, and then generate an intensity-adaptive mask for each branch based on a preset intensity threshold. That is:

[0112] .

[0113] The enhanced visible light and infrared features are then projected onto another branch to obtain the corresponding compensated features. and ;Right now:

[0114] .

[0115] Finally, the bimodal effective weights are synthesized based on the masks of the two branches. Visible light effective weight only and infrared effective weights only This leads to the hierarchical fusion features output by the soft-perception invalid mode gating mechanism. F out :

[0116] .

[0117] Example 4

[0118] Based on the solutions in Embodiments 1-3, this embodiment further provides a flame recognition device, which is essentially a computer device. It includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a trained flame detection model as described in any one of Embodiments 1-3 for flame detection of low-quality dual-spectral images. The flame detection model is used to generate detection results of flames contained in the input visible light image and infrared image.

[0119] In practical applications, the computer equipment can be a standalone computer device, such as a laptop, tablet, desktop computer, or a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers) capable of executing computer programs.

[0120] The computer device in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus. In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0121] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device.

[0122] Simulation test

[0123] To verify the performance of the flame detection method for low-quality bispectral images provided by this invention, technicians simulated and trained the relevant scheme and tested the performance of the trained network model.

[0124] I. Dataset, Model, and Training Strategy

[0125] Because existing technologies lack publicly available bimodal flame data containing low-quality samples from one side, this experiment independently captured visible light and near-infrared flame datasets. By adjusting camera parameters, the quality of some infrared or visible light images in the samples was reduced, even to the point of being completely black and invalid modes. The visible light and near-infrared bimodal flame dataset, captured using a dual-camera setup in the laboratory, has been aligned and contains over 15,000 data pairs. The original resolution was 5472×3648 pixels, but to facilitate subsequent work and reduce the training and inference burden, the image size was uniformly reduced to 800×509 pixels. 12,000 data pairs include annotations, while the remaining data are captured interference data. The training set and validation set were randomly split in a 2:1 ratio, with 9,627 data pairs in the training set and 4,491 data pairs in the validation set. When inputting into the network, the visible light images were 3-channel data, and the infrared images were 1-channel data.

[0126] The flame detection model in this experiment is an improvement upon the bimodal detection network YOLOv11n-midfusion, incorporating FCFM and / or SPMG as hierarchical fusion modules. No pre-trained model was used during training; instead, the constructed model was trained from scratch. After embedding FCFM and SPMG, hyperparameters for the bimodal branch of FCFM were designed, and hyperparameter experiments were conducted using an initial learning rate. During inference, the network weights that demonstrated the best performance in the aforementioned hyperparameter experiments were used. The input consisted of two images, and the detection results were presented in both images. When a detection was made in one modality, it was mapped to the same location in the other modality.

[0127] This experiment uses the average precision (mAP), a commonly used metric in object detection, as the evaluation standard. The intersection-over-union (IoU) threshold ranges from 50 to 95, displaying Map50 and Map50-95 metric data. It also measures method complexity in terms of parameter count. The learning rate is the default 0.01, free blending precision is not enabled, and the training epoch is 400.

[0128] II. Experimental Results and Analysis

[0129] 2.1 Considering that the proposed solution is geared towards flame detection and is intended for future migration to edge devices, this experiment uses a network with a scaling factor of 1 / 4 for both the baseline model and the improvements. To ensure fairness in performance comparison, this experiment compares the proposed solution with several network models (VI, IR, Baseline-n, HyperYolo-midfusion-n, PicoDet-midfusion-s, YOLOV11-CFT-n, YOLOV12-midfusion-n) that have similar computational complexity and parameter count. The experimental results are shown in Table 1.

[0130] Table 1: Performance Comparison of the Invention and the Baseline Network

[0131]

[0132] Analysis of the above data shows that the solution of the present invention effectively improves the performance of the model without increasing the number of parameters.

[0133] 2.2 Visualization

[0134] This experiment first visualizes the heatmaps of the enhanced feature images of FCFM in bright and dark scenes, and the results are as follows: Figure 8 As shown in the image, the left half represents a bright scene, and the right half represents a dark scene. The heatmap reveals that in bright scenes, FCFM enables the model to more precisely select and locate flame features, while also removing redundancy to some extent. In dark scenes, FCFM allows the model to focus more on learning the shape and edge features of objects in lower brightness, and also pays more attention to the features of the flame region.

[0135] This experiment further compares the feature images after the first layer of fusion using common two-modal feature fusion methods: channel concatenation (Concat), feature addition (ADD), and cross-modal fusion (CFT), with the feature images after fusion using the scheme of this invention (FCFM). The results are as follows. Figure 9 As shown in the figure, analysis of the data reveals that the method of this invention retains important complementary features of the two modes after fusion, while paying more attention to the target region and the edge of the target shape, which is beneficial for flame detection.

[0136] 2.3 Ablation Experiment

[0137] This experiment was designed as an ablation experiment to analyze the contributions of FCFM and SPMG introduced into the flame detection network. The results of the ablation experiment are shown in Table 2 below:

[0138] Table 2: Ablation Experiment Results of the Scheme of the Invention

[0139]

[0140] Analysis of the data in the table above shows that both FCFM and SPMG mechanisms can significantly improve the prediction accuracy of the model, and the performance improvement is even more pronounced when both mechanisms are used simultaneously.

[0141] 2.4 Hyperparameter Optimization

[0142] This experiment further analyzes the filtering radius of the high-pass and low-pass filters in FCFM. The impact of different values ​​on model performance was investigated, resulting in the results shown in Table 3 below:

[0143] Table 3: Performance Comparison of FCFM at Different Filtration Radius

[0144]

[0145] Analysis of the data in the table above shows that in the scheme containing a three-layer hierarchical fusion module, the best feature enhancement effect can be achieved when the filtering radii of the high-pass filter and the low-pass filter are set to 10 and 20, respectively.

[0146] This experiment further uses enhanced high-frequency feature EH and enhanced low-frequency information EL as the filtering direction operation representation to analyze their impact on model performance, and the experimental results are shown in Table 4 below:

[0147] Table 4: Performance Comparison of Different Enhancement Directions in FCFM

[0148]

[0149] Analysis of the data in the table above shows that, in order to obtain better feature enhancement effects, the visible light channel tends to enhance high-frequency features, while the infrared channel tends to enhance low-frequency features. When all features are enhanced, either high-frequency or low-frequency features, or both high-frequency infrared features and low-frequency visible light features are enhanced, the resulting feature enhancement effect is relatively poor.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A flame detection method suitable for low-quality dual-spectral images, characterized in that, It includes: A flame detection model is constructed, comprising a dual-branch backbone network, a hierarchical fusion module, a neck network, and a prediction head. The hierarchical fusion module includes a frequency domain complementary fusion mechanism (FCFM) and a soft-sensing invalid mode gating mechanism (SPMG) introduced in a specified layer of the dual-branch backbone network. The dual-branch backbone network is used to extract visible light and infrared features at different scales. FCFM uses an adjustable threshold high-pass filter to perform high-pass filtering on visible light features in the frequency domain, and uses an adjustable threshold low-pass filter to perform low-pass filtering on infrared features in the frequency domain. Then, the filtered features are residually concatenated with the original features, and then subjected to 1×1 convolution, batch normalization and SiLU activation to obtain the enhanced features of the corresponding branches; The enhanced features from the two branches are concatenated to form bisexual features at the corresponding scale. F dual ; SPMG is used to first calculate the global average intensity of the enhanced visible light and infrared features separately, and then generate an intensity-adaptive mask for each branch based on a preset intensity threshold; finally, the enhanced visible light and infrared features are projected onto the other branch to obtain the corresponding compensation features. and The effective weights for the two modes are synthesized based on the masks of the two branches. Visible light effective weight only and infrared effective weights only This leads to the hierarchical fusion features output by the soft-perception invalid mode gating mechanism. F out : ; The neck network is used to perform multi-scale fusion of the input features from various levels; the prediction head is used to output the prediction result based on the output features of the neck network. The flame detection model is trained using a sample dataset containing low-quality visible light images or low-quality infrared images, and the trained flame detection model is then used to detect flames on the input samples.

2. The flame detection method for low-quality dual-spectral images as described in claim 1, characterized in that: The high-pass filter and low-pass filter respectively describe the temporal characteristics of visible light features and infrared features according to the channel. f ( x,y Perform a two-dimensional discrete Fourier transform to obtain the corresponding frequency domain features. F ( u,v Then, filtering is performed; finally, the time-domain features are obtained through inverse Fourier transform. And / or, the expression for the two-dimensional discrete Fourier transform is as follows: ； In the above formula, ( x,y () represents the time-domain coordinates of the feature map; u,v () represents the frequency domain coordinates of the feature map; H and W represent the height and width of the feature map; e Represent natural numbers; i It represents the imaginary unit.

3. The flame detection method for low-quality dual-spectral images as described in claim 2, characterized in that: The expression for the high-pass filter is: ； In the above formula, This represents the visible light characteristics after high-pass filtering; Indicates the backbone network number i Visible light characteristics of the layer; Indicates the first i The filter radius of the layer; and These represent the two-dimensional discrete Fourier transform and its inverse transform, respectively; This represents the mask generated by the high-pass filter; Real(·) represents element-wise multiplication; Real(·) represents the real part extraction operation. The expression for the low-pass filter is: ； In the above formula, This indicates the infrared characteristics after low-pass filtering. Indicates the backbone network number i Infrared characteristics of the layer; Indicates the first i The filter radius of the layer; This represents the mask generated by the low-pass filter.

4. The flame detection method for low-quality dual-spectral images as described in claim 3, characterized in that: and Here are a set of hyperparameters to be optimized in the frequency domain complementary fusion mechanism; their feasible regions are as follows: ； In the above formula, H i and W i These represent the backbone network's... i The height and width of the visible light / infrared features of the layer.

5. The flame detection method for low-quality dual-spectral images as described in claim 3, characterized in that: Hierarchical fusion features of FCFM output The expression is: ； In the above formula, Concat represents the feature concatenation operation; BN represents batch normalization; Conv 1×1 Represents a 1×1 convolution; Indicates the SiLU activation function; and They represent the characteristics of visible light, respectively. and infrared features Enhanced features; and These represent the visible light characteristics after high-pass filtering in the frequency domain and the infrared characteristics after low-pass filtering in the frequency domain, respectively.

6. The flame detection method for low-quality dual-spectral images as described in claim 1, characterized in that: arbitrary mode m Feature map F m ( c , h , w global average intensity I m The calculation formula is as follows: ； In the above formula, C, H, and W represent the number of channels, height, and width of the feature map, respectively; m=vis and m=nir The times represent the visible light mode and the infrared mode, respectively.

7. The flame detection method for low-quality dual-spectral images as described in claim 6, characterized in that, based on I m Generate a mask for the corresponding mode. M m The expression is as follows: ； In the above formula, This represents the Sigmoid activation function; This indicates the preset intensity threshold; k This indicates the preset soft-gating slope.

8. The flame detection method for low-quality dual-spectral images as described in claim 1, characterized in that: The formulas for generating projections of visible light features and infrared features are as follows: ； In the above formula, P represents the projection layer; Represents the visible light characteristic F vis Compensation features projected onto infrared modes; Indicates infrared feature F nir Compensation features projected onto the visible light mode.

9. The flame detection method for low-quality dual-spectral images as described in claim 1, characterized in that, Bimodal effective weights Visible light effective weight only and infrared effective weights only The formula for generating it is as follows: ； In the above formula, M vis and M nir These represent visible light mask and infrared mask, respectively.

10. A flame identification device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it creates a trained flame detection model for the flame detection method for low-quality bispectral images as described in any one of claims 1-9; the flame detection model is used to generate detection results of flames contained in the input visible light image and infrared image.

Images