Flame detection methods and equipment based on flame specificity and multi-exposure image fusion

Abstract

This invention relates to the field of image recognition, specifically a flame detection method and device based on flame specificity and multi-exposure image fusion. The flame recognition model constructed by this method includes an image fusion module (EMEIF) and a flame detection network. The image fusion module generates a fused image based on multiple consecutive multi-exposure images, and the flame detection network generates the flame recognition result based on the fused image. The EMEIF module consists of an encoder, a fusion unit, and a decoder; the encoder extracts features from the input image; the fusion unit uses adaptive fusion weights to weight and fuse the extracted feature information; the fused features are processed by the decoder to obtain the fused image. For the newly designed network model, this invention introduces a three-stage training strategy, allowing the model to transition from learning the features of flame data captured in a baseline image to learning the features of flame data after multi-exposure fusion. This invention overcomes the shortcomings of existing solutions in terms of detection accuracy and cost.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to a flame detection method based on flame specificity and multi-exposure image fusion, as well as the corresponding computer program product, storage medium and computer device. Background Technology

[0002] Traditional solutions mostly use temperature sensors, smoke sensors, etc. to detect and warn of fires. However, such solutions have a high false alarm rate and cannot guarantee real-time performance. To solve this problem, a series of computer vision-based methods have been proposed and applied to fire detection. Among them, the YOLO series is widely used because of its lightweight and fast response speed. Its main method is to put the labeled flame dataset into the model. The model learns the features of the flame through layers of network to achieve the detection and recognition of the flame. However, although such solutions perform well in experiments, they are not ideal in real-world detection. The reasons for this situation are as follows: (1) The dataset of flame images is relatively scarce. At present, most fire experiments use only a few thousand images in the dataset, which lacks sample data covering various interference sources (such as car lights, glass reflections, etc.). In real-world use, the images captured by the camera contain many interference sources, which makes it easy for the model to make false judgments. (2) The sample images in actual detection are easily affected by environmental factors, resulting in overexposure and underexposure. The existing network models are not good at recognizing flame targets in these two extreme low-quality images.

[0003] Considering the high cost of expanding flame datasets, researchers have proposed a series of methods to improve network detection performance based on small sample sizes. Among them, Liu Jiali et al. proposed a flame detection method using multi-information fusion in the Ohta color space. This method analyzes the saturation of the red component in the flame color space, ensuring its saturation is greater than a given red component threshold RT. It also sets a threshold for flame saturation to avoid the influence of background lighting, which improves detection accuracy to some extent. Tang Linfeng et al. introduced illumination factors into the model, allowing it to learn the difference between day and night, and controlling the soft selection of visible light and infrared light by changing weights. However, this approach did not significantly improve performance when introducing images with extreme flame exposure conditions. While numerous improvement methods have indeed improved flame detection accuracy, few have effectively handled extreme image exposure conditions. Furthermore, most improvement schemes still cannot effectively overcome the interference source problem in visible light flame image detection.

[0004] In complex target detection tasks, researchers have proposed image fusion detection, which can be broadly categorized into three types: early-stage fusion, mid-stage fusion, and late-stage fusion. Mid-to-late-stage fusion shows the most significant improvement. However, mid-to-late-stage fusion often requires dual-branch or even multi-branch backbone networks, leading to a trade-off between model time and accuracy. Early-stage fusion, which aims to fuse visible light and infrared images before inputting the image into the backbone network, hinders the interaction of visible light and infrared image features, resulting in inferior performance compared to mid-to-late-stage fusion. Inspired by pedestrian image fusion detection, flame detection has also seen the emergence of images capable of fusing light and infrared images. Flames have gained significant attention due to their strong saliency in infrared images and the ability to eliminate certain interference sources. However, flame datasets are scarce, and corresponding infrared datasets are even more limited. Even with datasets that consume substantial resources to capture visible light and infrared images simultaneously, data annotation, processing, and alignment remain challenging, and the models are costly, requiring cameras capable of capturing both visible light and infrared images simultaneously. Even with significant detection results, widespread adoption is impractical. Summary of the Invention

[0005] To address the issues of high accuracy and cost in existing flame recognition technologies based on the fusion of infrared and visible light images, this invention provides a flame detection method based on flame specificity and multi-exposure image fusion, along with corresponding computer program products, storage media, and computer equipment.

[0006] The technical solutions provided by the present invention are as follows:

[0007] A flame detection method based on flame specificity and multi-exposure image fusion includes the following steps:

[0008] A flame recognition model is constructed, comprising an image fusion module (EMEIF) and a flame detection network. In the flame recognition model, the image fusion module is used to determine the flame detection network based on the image fusion module's characteristics. n A fused image is generated from a series of consecutive multi-exposure images. A flame detection network is then used to generate flame recognition results based on this fused image. The image fusion module consists of an encoder, a fusion unit, and a decoder. The encoder extracts features from each of the input multi-exposure images, obtaining feature maps F1~F2. n The fusion unit uses preset fusion weights W1~W1 based on relative exposure. n For F1 to F respectively n After weighting, channel stitching is performed to obtain fused features. The fused features are then processed by a decoder to obtain the fused image.

[0009] The initial model was selected from EMEIF and a pre-trained flame detection network; it was trained as follows: (1) Setting up a loss that includes inter-image structural similarity. L SSIM and weight loss L wight loss function L 1. Train the initial model once and iteratively update the weight coefficients of the fusion units in EMEIF. (2) Set the regression loss to include bounding boxes. L box Target confidence loss L obj Category classification loss L cls joint losses L 2. Perform a second round of training on the model after the previous round of training; iteratively update the model parameters of the flame detection network. (3) Set the parameters to include L 1. L box , L obj and L cls joint losses L 3. Perform three rounds of training on the model after the previous round of training, and iteratively update the model parameters of EMEIF and the flame detection network;

[0010] Multiple consecutive multi-exposure images are input into a flame recognition model that has been trained for three rounds to achieve flame recognition.

[0011] As a further improvement of the present invention, the flame detection network adopts a pre-trained network based on YOLOv5.

[0012] As a further improvement of this invention, the encoder in the image fusion module employs a P-DenseBlock encoder. The P-DenseBlock encoder sequentially includes a first convolutional layer PConv using a 3×3 windmill-shaped convolutional module, and third, fourth convolutional layers DPC1, DPC2, and DPC3 using 3×3 depthwise separable convolutions. The output of each convolutional layer in the P-DenseBlock encoder is residually concatenated with the outputs of subsequent convolutional layers.

[0013] As a further improvement of the present invention, the decoder in the image fusion module adopts the DenseNet decoder, which contains four convolutional layers with 3×3 kernels in sequence.

[0014] As a further improvement of the present invention, in the fusion unit of the image fusion module, the fusion weights W1~W n The generation method is as follows:

[0015] S01: Calculate the feature map extracted from the original image at each exposure level using the following formula. F i Exposure value E i :

[0016] ;

[0017] In the above formula, I ( p ) represents the first element in the input image. p The normalized intensity value of each pixel; N This represents the total number of pixels in the image.

[0018] S02: The exposure values ​​of each image are calculated using the following formula. E i Convert to relative exposure R i :

[0019] .

[0020] S03: The relative exposure is calculated using the following non-linear function. R i Convert to dynamic weights W i :

[0021] ;

[0022] In the above formula, and Let be two learnable weight coefficients in the fusion unit; where, increasing This will significantly increase the weight of low-exposure images, while the weight of high-exposure images will approach 0; The threshold for relative exposure can be adjusted, affecting the average point of weight distribution.

[0023] As a further improvement of the present invention, one round of training uses a dataset consisting of real sample images.

[0024] The second round of training uses a dataset composed of real sample images and fused images from the EMEIF output of the first round of training.

[0025] The three rounds of training used a dataset consisting of real sample images and their augmented images.

[0026] As a further improvement to the present invention, the loss function L The expression for 1 is as follows:

[0027] ;

[0028] In the above formula, I i Indicates the input number of the first... i The original image at this exposure level; Represents a fused image; This is an evaluation function used to measure the structural similarity between the fused image and the input image; Indicates the first i Original image structure adjustment factor under various exposure levels; Used to calculate the square of the L2 normal form for each weight; Represents a parameter factor. =0.0001.

[0029] As a further improvement to the present invention, the loss function L The expression for 2 is as follows:

[0030] L 2 =L box + L obj + L cls .

[0031] As a further improvement to the present invention, the loss function L The expression for 3 is as follows:

[0032] L 3 =ρL 1 +L box + L obj + L cls ;

[0033] In the above formula, ρ Represents the balance factor. ρ =0.1.

[0034] As a further improvement of the present invention, the original input of the flame recognition model is three consecutive multi-exposure images; during the inference stage, the three consecutive multi-exposure images are continuously acquired by the camera at a sampling interval of 0.2s, and the exposure of each image is adjusted by changing the exposure duration.

[0035] The present invention also includes a computer program product comprising a computer program that, when executed by a processor, creates a trained flame recognition model as described in the aforementioned flame detection method based on flame specificity and multi-exposure image fusion, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

[0036] The present invention also includes a storage medium storing a computer program, which, when executed by a processor, creates a trained flame recognition model as described in the aforementioned flame detection method based on flame specificity and multi-exposure image fusion, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

[0037] The present invention also includes a computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it creates a trained flame recognition model as described in the aforementioned flame detection method based on flame specificity and multi-exposure image fusion, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

[0038] The present invention has the following beneficial effects:

[0039] Based on the specificity of flame detection, this invention constructs a novel flame recognition model comprising an image fusion module and a flame detection network, and uses continuously sampled multi-exposure images to achieve flame recognition. The image fusion module in this method can adaptively fuse multiple images with different exposures over a short time difference. The resulting fused image effectively integrates feature information from different exposure images, improving dark and bright areas in the original image, providing the flame detection network with more suitable data for flame detection, effectively overcoming the impact of underexposed and overexposed images on detection accuracy, and avoiding misjudgments when the network detects images with extreme exposure.

[0040] To improve the training performance of the flame recognition model incorporating the EMFIF module, this invention employs a unique three-stage training strategy. This strategy allows the model to transition from learning features from baseline flame data to learning features from multi-exposure fusion flame data. Specifically, the initial multi-exposure fusion module is a novel method for expanding the flame dataset, and its phased mixed data training further enhances the generalization ability of the flame detection network.

[0041] The image fusion strategy provided by this invention is a pre-processing fusion strategy, which can fuse multi-exposure images captured continuously by a single camera to obtain a fused image with richer information and suitable for flame target detection. This overcomes the problem that the amount of information contained in the sample images is limited in traditional algorithms that adjust the image exposure in post-processing. Attached Figure Description

[0042] Figure 1 This is a network architecture diagram of the flame recognition model provided in Embodiment 1 of the present invention.

[0043] Figure 2This is a schematic diagram of the image fusion module in the flame recognition model of Embodiment 1 of the present invention.

[0044] Figure 3 The module structure of the encoder part of the image fusion module in Embodiment 1 of the present invention is shown.

[0045] Figure 4 This is the module structure of the decoder part of the image fusion module in Embodiment 1 of the present invention.

[0046] Figure 5 This study aims to test two typical sets of flame data taken at different times and distances during the experiment, as well as images obtained using the pyramid fusion method.

[0047] Figure 6 For testing experiments Figure 5 Flame detection results from two fused images.

[0048] Figure 7 These are some examples of missed and false detections in the test experiment.

[0049] Figure 8 This is a partial image fusion effect diagram of the model trained by the scheme of the present invention in the test experiment. Detailed Implementation

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0052] Example 1

[0053] Image fusion techniques are commonly used in image-based tasks such as target recognition, classification, and image segmentation. Image fusion refers to integrating information from multiple source images (potentially from different sensors, viewpoints, and time points) into a single output image. This process aims to generate a comprehensive image that contains all key details and has enhanced properties by effectively merging complementary, redundant, or unique information from each source image. Broadly speaking, image fusion can be categorized into pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion is unsuitable for target detection due to its high computational cost, demanding hardware requirements, and susceptibility to pixel contamination. With increasing investment in machine learning, feature-level fusion after feature extraction using convolutional neural networks has become mainstream, leading to the development of multimodal target detection. Multimodal target detection is mainly divided into early-stage fusion, mid-stage fusion, and late-stage fusion. Mid- and late-stage fusion primarily involves feature extraction via dual-branch architectures followed by feature fusion at the backbone network or at the network neck, mainly to improve the network's learning ability. Although most experiments have shown that early-stage fusion methods are less effective than mid-to-late-stage fusion methods, early-stage fusion mainly aims to provide data with richer features. However, due to the presence of more interfering information, it performs poorly on some target detection tasks.

[0054] In their research on flame detection, the technicians in this embodiment discovered that during flame image capture, excessive brightness or overexposure can lead to highlights in certain areas of the image, resulting in a loss of detail and color features, and the appearance of numerous interference sources (such as glass reflections and white walls). Furthermore, because flames themselves possess high brightness, obtaining high-quality flame data is challenging in various environments. For example, when capturing flame data in a fire laboratory, the presence of significant refracted light indoors easily leads to overexposure. Additionally, when capturing flame data outdoors under strong sunlight, the high brightness weakens the color features of the flame, and the texture details at the flame edges are easily blurred, easily resulting in poor image quality and potential model misjudgment.

[0055] To address the aforementioned quality issues in visible light images of flames, fusing visible light and infrared images can solve the problem to some extent. However, when the visible light image itself is overexposed and the flame's color and texture features are weak, the fusion model tends to prioritize infrared, which primarily focuses on the flame's shape features. Furthermore, in real-world shooting, it has been found that when visible light is overexposed, large areas of near-infrared images are also overexposed. Deploying edge devices (such as cameras) aligned with visible and infrared images is also costly. Additionally, since mid-to-late-stage fusion mainly involves stitching together features passed through the backbone network before detection, it cannot effectively handle overexposure interference during the fusion stage.

[0056] Furthermore, the technicians in this embodiment also discovered that flame data exhibits certain specificities in target detection. This specificity allows for excellent adaptability of the initial fusion process in the flame domain. Specifically, compared to conventional dynamic target detection tasks (such as pedestrian and vehicle detection), this specificity in the flame recognition domain manifests as follows:

[0057] Pedestrian and vehicle target detection is achieved using video stream data acquired by cameras. The processing involves frame extraction from the video stream and detection using each frame. Since the targets in this task are dynamic, existing multi-exposure fusion methods must ensure image alignment and consistent shooting times when generating new images to guarantee consistent coverage of pedestrians and vehicles after fusion. For example, inconsistent shooting times can lead to partial overlap of pedestrian images after fusion, resulting in significant differences between the pedestrians and those learned by the detection network, causing the network to fail to make predictions or lower its confidence level. Therefore, for moving targets, multi-exposure fusion relies on post-processing exposure adjustment algorithms to adjust the original image to obtain images with different exposures within the same scene. However, the image quality obtained by adjusting the original image to obtain images with different exposures is generally poor, with information loss.

[0058] Based on this, the technicians of this embodiment noted that flame shapes can change rapidly due to numerous reasons, and fire datasets themselves are scarce. However, fire detection models using smaller datasets can still identify irregularly shaped flames under appropriate exposure conditions. This indicates that existing flame target detection networks have strong generalization ability regarding flame shapes. The technicians of this embodiment therefore speculate that this may be because, in the view of the flame target detection network, flames are considered to be fluid like water; therefore, mixing flames with different shapes from different images still results in a single flame.

[0059] Based on the findings regarding the impact of exposure and morphology fusion on flame target detection accuracy, this embodiment proposes a flame detection method based on flame specificity and multi-exposure image fusion. This method directly acquires multiple consecutive visible light images with different exposures using hardware devices such as cameras during the image acquisition stage. Then, it performs image fusion on these consecutive images in the initial stage. Finally, a flame detection network identifies the flame based on the fused image. This novel approach is expected to reduce detection costs while effectively mitigating the impact of overexposure in various environments on flame detection accuracy. Specifically, the flame detection method provided in this embodiment includes the following process:

[0060] I. Construction of Flame Recognition Model

[0061] First, this embodiment utilizes an existing flame detection network in conjunction with a newly designed image fusion module EMEIF to construct a network as follows: Figure 1 The diagram illustrates the creation of a flame recognition model. The flame detection network can be based on a YOLOv5 network model, or it can employ other network models with different architectures that offer high flame recognition performance, such as DERT, Faster R-CNN, and RetinaNet. Figure 1 In the flame recognition model, the image fusion module receives multiple (assuming the number is n, n≥2) consecutive multi-exposure images and fuses them to obtain a single fused image. The fused image generated by the image fusion module is input into the flame detection network, which then identifies the flame based on the fused image.

[0062] Specifically, in this embodiment, the continuous images input to the image fusion module should be multiple flame images taken by the same camera at a fixed position with different exposures. Considering that during the shooting stage, image brightness is mainly related to exposure and ISO, exposure is the amount of light received by the photosensitive element during the exposure time. Controlling exposure is one of the main ways a camera controls light; exposure equals the product of the speed at which the photosensitive element receives light and the exposure time (shutter time). When the exposure is constant, the brightness of the image can be increased by increasing the ISO value of the photosensitive element. However, increasing ISO easily generates noise, so generally, the exposure is adjusted to increase image brightness. Overexposure refers to images captured with excessively high exposure. For the specific problem of flame detection, this embodiment uses the camera's automatic exposure (AE) to select the exposure range and changes the brightness of each image by adjusting the exposure time of each captured image. Furthermore, for multiple images in the same detection task, this embodiment also sets the time interval between multiple images with different exposures to within 1 second to prevent excessive differences in flame shape that would hinder fusion and detection.

[0063] Furthermore, the number of multi-exposure images input into the flame recognition model should not be excessive. This is because the most effective flame-related image information is usually contained within a narrow exposure range. Introducing too many images at different exposures may result in too much interference information in the fused image, which is detrimental to detection. Therefore, in a typical scheme of this embodiment, the original input of the flame recognition model is three consecutive multi-exposure images; during the inference phase, the three consecutive multi-exposure images are continuously acquired by the camera at a sampling interval of 0.2 seconds, and the exposure of each image is adjusted by changing the exposure duration.

[0064] To achieve efficient fusion of multiple images with different exposures, the image fusion module designed in this embodiment is as follows: Figure 2As shown, it consists of three parts: an encoder, a fusion unit, and a decoder. The encoder extracts features from the input multi-exposure images, obtaining feature maps F1~F2. n The fusion unit is used to determine the fusion weights W1~W1 based on the preset relative exposure. n For F1 to F respectively n After weighting, the individual feature images are then concatenated channel by channel to obtain the fused features. Finally, the fused features output by the fusion unit are processed by the decoder to obtain the fused image. .

[0065] In practical applications, there are many encoder options. Considering that the EMEIF module uses multi-branch fusion during training, it needs to preserve contextual information as much as possible while ensuring inference speed. Traditional convolutions have limited receptive fields, making it difficult to capture all information. The encoder used in this embodiment replaces the ordinary convolution (Conv) in the DenceBlock encoder with a windmill-shaped convolution (PConv), expanding the receptive field with minimal increase in parameters. To distinguish it from the original encoder, this embodiment names it P-DenseBlock.

[0066] like Figure 3 As shown, the P-DenseBlock encoder sequentially includes a first convolutional layer PConv using a 3×3 windmill-shaped convolutional module, and a second convolutional layer DPC1, a third convolutional layer DPC2, and a fourth convolutional layer DPC3 using a 3×3 depthwise separable convolutional module. In this embodiment of the P-DenseBlock encoder, the number of input channels and output channels of the first convolutional layer are 1 and 16, respectively; the number of input channels and output channels of the second convolutional layer are 16 and 16, respectively; the number of input channels and output channels of the third convolutional layer are 32 and 16, respectively; and the number of input channels and output channels of the fourth convolutional layer are 48 and 16, respectively. Furthermore, the output of each convolutional layer in the P-DenseBlock encoder is residually concatenated with the outputs of subsequent convolutional layers. Therefore, the feature maps F1~F2 after encoder processing are... n The number of channels is 64 for each.

[0067] Based on this, the fusion unit uses preset fusion weights W1~W n For F1~F n After weighted summation and channel concatenation, the number of channels for the fused features is also 64. In the fusion stage, this embodiment designs an adaptive weighting mechanism related to image exposure to adaptively fuse the three features, ensuring that the fused image obtained after decoding contains the most suitable feature information for flame detection.

[0068] In this context, considering that brightness information and other data are allocated to image features after the image has been encoded, simply stitching or adaptively fusing these features may not yield suitable brightness information for flame detection. For flames, which inherently possess high brightness, lower exposure is more suitable for detection. In low-exposure images, because flames themselves have high brightness, they may actually gain more texture features due to reduced exposure, even when other objects lose texture and shape details. Therefore, the image fusion module in this embodiment tends to lower the exposure of the fused image, but not so low that it only retains flame features; otherwise, training with the obtained data would reduce its robustness. Thus, it is crucial to ensure that scene details are not lost as much as possible. To address this issue, this embodiment introduces fusion weights W1~W1 based on relative exposure in the image fusion module. n .

[0069] Specifically, in the fusion unit of the image fusion module provided in this embodiment, the designed fusion weights W1~W n The generation method is as follows:

[0070] S01: Calculate the feature map extracted from the original image at each exposure level using the following formula. F i Exposure value E i :

[0071] ;

[0072] In the above formula, I ( p ) represents the first element in the input image. p The normalized intensity value of each pixel. I ( p )∈[0,1]; N This represents the total number of pixels in the image.

[0073] S02: The exposure values ​​of each image are calculated using the following formula. E i Convert to relative exposure R i :

[0074] .

[0075] Analyzing the above formula reveals that, for any given image, when its corresponding exposure value... E i The higher the value, the higher the relative exposure obtained after processing with the above formula. R iThe lower the value, the better. Specifically, the relative exposure value of the highest exposure image is 0, which ensures that low-exposure images receive a higher score during the fusion process in this embodiment.

[0076] S03: The relative exposure is calculated using the following non-linear function. R i Convert to dynamic weights W i :

[0077] ;

[0078] In the above formula, and These are two learnable weight coefficients in the fusion unit. Where, increasing... This will significantly increase the weight of low-exposure images, while the weight of high-exposure images will approach 0; The threshold for relative exposure can be adjusted, affecting the average point of weight distribution.

[0079] Therefore, it can be seen that in the generation mechanism designed in this embodiment, the fusion weights W1~W n The value of is adaptively adjusted based on the relative exposure of the original images input to the network model, and is also influenced by learnable weight coefficients. and The impact of this is that, in subsequent processes, the image fusion model in the network model can be specifically trained to obtain the optimal weight coefficient values; this ensures the recognition accuracy of the network model for multi-exposure images in different scenarios, thereby improving the generalization ability of the solution.

[0080] In the decoder section, considering that the encoder already has multiple branches and some parameters have been added, and based on the need for lightweight design, this embodiment adopts the DenseNet decoder. For example... Figure 4 As shown, the DenseNet decoder contains four convolutional layers with 3×3 kernels. The first convolutional layer has 64 input channels and 64 output channels; the second has 64 input channels and 32 output channels; the third has 32 input channels and 16 output channels; and the fourth has 16 input channels and 1 output channel. Therefore, the image fusion module in this embodiment can output a fused image containing rich information from the original images at different exposures, with a total of 1 channel.

[0081] II. Training of the Flame Recognition Model

[0082] In this embodiment, to ensure that the newly designed flame recognition model achieves optimal target detection performance, a corresponding multi-stage training strategy is also designed for it. Specifically, this embodiment selects an initial model consisting of an EMEIF module and a pre-trained flame detection network. The flame detection network can use various existing pre-trained network models and retain their model parameters. In the EMEIF module, this embodiment sets the weight coefficients... The initial setting is 0.6. The initial value is 0.5.

[0083] In practical applications, this embodiment designs a three-stage training strategy. The first stage primarily trains the EMEIF module, the second stage updates the model parameters of the flame detection network, and the third stage trains the entire flame recognition model, including both the EMEIF module and the flame detection network. The specific training process is as follows:

[0084] (1) Set up a loss function that includes structural similarity between images. L SSIM and weight loss L wight loss function L 1. Perform one round of training on the initial model, iteratively updating the weight coefficients of the fusion units in EMEIF. This first round of training uses a dataset consisting of real sample images; the loss function used in this stage is... L The expression for 1 is as follows:

[0085] ;

[0086] In the above formula, I i Indicates the input number of the first... i The original image at this exposure level; Represents a fused image; This is an evaluation function used to measure the structural similarity between the fused image and the input image; Indicates the first i Original image structure adjustment factor under various exposure levels; Used to calculate the square of the L2 normal form for each weight; In this embodiment, a parameter factor is represented. =0.0001.

[0087] (2) Set the regression loss to include bounding boxes L box Target confidence loss L obj Category classification loss L cls joint lossesL 2. Perform a second round of training on the model after the previous round of training; iteratively update the model parameters of the flame detection network. The second round of training uses a dataset consisting of fused images of real sample images and the EMEIF output from the first round of training. The loss function used in this stage is... L The expression for 2 is as follows:

[0088] L 2 =L box + L obj + L cls .

[0089] (3) Setting includes L 1. L box , L obj and L cls joint losses L 3. Perform three rounds of training on the model after the previous round of training, iteratively updating the model parameters of EMEIF and the flame detection network. The three rounds of training use a dataset consisting of real sample images and their augmented images. The sample images obtained through data augmentation include both new sample images obtained by cropping the original images and new sample images reconstructed by combining the fused image obtained in the first stage with two of the original sample images. The loss function for this stage... L The expression for 3 is as follows:

[0090] L 3 =ρL 1 +L box + L obj + L cls ;

[0091] In the above formula, ρ Represents the balance factor. ρ =0.1.

[0092] III. Application of Flame Recognition Model

[0093] After the three rounds of training described above, a flame recognition model that meets the requirements for both detection accuracy and robustness can be obtained. This embodiment saves the parameters of the network model with the best performance after verification. Then, in practical applications, multiple consecutive multi-exposure images are input into the flame recognition model after three rounds of training. The model then performs inference and identifies the flames contained within them.

[0094] Example 2

[0095] The flame detection method based on flame specificity and multi-exposure image fusion provided in Example 1 is essentially a data processing method. In order to better apply the scheme in Example 1, this example further provides a computer program product, storage medium and computer device that can implement the method.

[0096] Specifically, the computer program product provided in this embodiment includes a computer program. When the computer program is executed by a processor, it creates a trained flame recognition model as in the flame detection method based on flame specificity and multi-exposure image fusion in Embodiment 1, thereby enabling the identification of flames contained in multiple consecutive multi-exposure images.

[0097] The storage medium provided in this embodiment stores a computer program. When the computer program is executed by the processor, it creates a trained flame recognition model as in the flame detection method based on flame specificity and multi-exposure image fusion in Embodiment 1, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

[0098] The computer device provided in this embodiment includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it creates a trained flame recognition model as in the flame detection method based on flame specificity and multi-exposure image fusion in Embodiment 1, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

[0099] In practical applications, this computer device can be embedded and deployed in terminal devices with fire monitoring capabilities to support the aforementioned data processing. It can also be used as a standalone computer to support data processing needs for flame detection in multi-exposure images acquired from the front end in certain scenarios. This non-embedded computer device can be a medium to large-sized computer, such as a laptop, tablet, desktop computer, or a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers) capable of executing computer programs.

[0100] Specifically, the computer device in this embodiment includes, but is not limited to, a memory and a processor that can be interconnected via a system bus. In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. Furthermore, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0101] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device.

[0102] Test Experiment

[0103] I. Theoretical Verification of the Strategy in the Invention

[0104] 1. Verify the impact of flame morphology on detection accuracy

[0105] This experiment first pre-trained a YOLOv5 network using a dataset of real flames, resulting in a YOLOv5-based flame detection network. Then, approximately 20 minutes of video footage of a flame burning in a laboratory was captured, with exposure controlled to ensure clarity. During filming, the program captured one image every 0.3 seconds, resulting in 1000 consecutive images. Two flame images spaced 0.3 seconds apart were paired and fused using a pyramid fusion method, yielding 500 fused flame images. The flame shape in these fused images differed slightly from the original images due to the time interval. These 500 images were then fed into the initially trained flame detection network, and the network still accurately identified the flames with relatively small differences in confidence. Subsequently, the experiment re-filmed the network using three images spaced 0.6 seconds apart, fused together, and fed into the flame detection network again; the model still successfully identified the flames.

[0106] Figure 5 part (a) and Figure 5 Part (b) shows two sets of flame data with the same exposure, taken at different times and distances, and their images obtained using the pyramid fusion method. The detection results obtained after detecting the flames using pre-trained flame detection weights are shown below. Figure 6 part (a) and Figure 6 As shown in part (b), the detection results under different conditions still have a high confidence level. Based on the above experiments, it can be confirmed that the detection network has strong generalization ability for the shape of the flame in the visible light flame target detection.

[0107] 2. The impact of image sampling method on detection accuracy

[0108] In this experiment, the time interval for image acquisition was varied and the number of images to be fused was modified. The resulting 3000 flame data images were then fed into YOLO v5 for training. Subsequently, the trained weights were used to detect the original flame data. The network successfully identified the flames, and the confidence level was not significantly different from that during training and validation.

[0109] Based on the experimental data, it was found that when the wind direction does not change significantly, the best image quality is obtained by fusing flame images within a 1-second interval. Fusing 3 flame images is the maximum number of images. Increasing the number of images beyond that will result in a significant difference between the fused image and the original image, and will also increase the computational load.

[0110] 3. The impact of overexposed sample images on detection accuracy

[0111] This experiment, using overexposed images for flame target detection, revealed that the network failed to identify flames regardless of whether they were fused or not. Furthermore, at certain exposure levels, fused flames or reflections were easily falsely detected. When the image exposure was too high, flame features were weakened, leading to either missed or false detections. Conversely, at excessively low exposures, while flame features were relatively clear, the weakened surrounding environmental features resulted in false detections of objects with similar colors, such as lights. Figure 7 It displays a series of cases of missed or false detections.

[0112] The above three experiments further confirm that flame target detection is specific, unlike pedestrian or vehicle detection which has high shape requirements and weak shape generalization. Flame target detection networks naturally possess strong generalization capabilities for the shape of flames within the detected target. Therefore, non-real-time image fusion on visible light images has minimal impact on detection accuracy. This demonstrates that the strategy proposed in this invention for flame detection using multi-exposure images acquired continuously over a short period is feasible and supported by theoretical evidence. Simultaneously, the experiments also confirm that for self-illuminating targets like flames, overcoming or adjusting exposure is necessary when designing the fusion module.

[0113] II. Performance Comparison Test between the Invention and Existing Solutions

[0114] 2.1 Model Training

[0115] This experiment is based on the preceding text. Figures 1-4 The proposed scheme constructs and trains a flame recognition model. To obtain a valid dataset, this experiment captured approximately 6,000 images of flames in relatively simple indoor scenes, where the surrounding scene remained static or underwent only slight changes, except for variations in the flames themselves. Furthermore, over 20,000 flame images were captured in complex outdoor scenes, including images of flames whose morphology changed under different wind conditions.

[0116] In the first round of training, this experiment divided the 27,456 data points into 9,152 groups of three, corresponding to 8,219 annotations. Each group of data used one set of annotations, and one-quarter of the data, or 2,288 groups, was used as the validation set. Approximately 2,000 groups of data from simple indoor scenes were selected and fed into the EMEIF module for pre-training to achieve the effect of pre-training, with 50 epochs.

[0117] In the second round of training, one data point was randomly selected from each of the 9152 original datasets as the baseline, resulting in 9152 data points. Then, 2288 data points were randomly selected from these 9152 datasets. Another 2288 data points were then randomly selected from the dataset obtained in the second stage of fusion. The results from both selections were mixed and shuffled, and one-quarter of the samples were used as the validation set. This set was then fed into YOLOv5s for training. The simplest data augmentation method was used: cropping. The epoch was 80, and the learning rate was 0.01. The primary goal was to train its flame detection capabilities in complex scenarios and improve its robustness.

[0118] In the third training phase, at least three images are selected for each scene dataset. The 9152 images obtained from the second training phase are used as one image in each scene dataset. Then, two images are extracted from each of the original 9152 datasets and merged into a new 9152 scene datasets, which are then fed into the entire network for training. The weights used in the fusion and detection modules are those obtained from the first two training phases. After the data passes through the fusion module, the resulting 9152 images are directly used to train the detection module for epoch 150, with the weights obtained from the third training phase taking precedence.

[0119] Figure 8 The image fusion effect of the model trained by the present invention is shown. The first three columns are three short-time-difference flame data with different exposures, with the exposures ranging from high to low. The last column, output, is the output image of the exposure fusion module of the present invention. Observing the image, it can be found that the solution of the present invention can fully solve the problem of environmental feature disorder caused by overexposure. It can retain some of the characteristics of the flame when overexposed while avoiding the weakening of environmental features when underexposed.

[0120] 2.2 Comparative Experiment:

[0121] To verify the advantages of the flame detection performance of the solution provided by this invention, the technicians conducted a comparative experiment. Since the flame dataset is relatively scarce and does not meet the experimental requirements of the solution for multiple exposures, this experiment uses a dataset taken by the technicians themselves for performance testing, and selects several existing multi-exposure fusion algorithms as control groups to compare the prediction performance with this solution.

[0122] (2.2.1) Data set.

[0123] Research in video fire detection has been ongoing for over two decades, but currently, there is no fire video database of suitable size, rich content, and standardized format available for researchers. The video and image data used by researchers mainly comes from four sources: First, several small, publicly available fire video databases, which generally have limited data volume and inconsistent sizes, formats, and scenes; second, fire video data collected from relevant fire protection agencies, such as local fire bureaus and fire research institutes. This part of the video data represents real fire records and is most suitable for video fire detection research, but currently, cooperation between relevant institutions is not close enough, and this part of the data remains to be explored; third, relevant fire video and image data is scraped from the internet. Images retrieved online using keywords such as "smoke" and "fire" vary widely in content, often being news illustrations or artistic photographs, which differ significantly from real fire images; fourth, researchers design their own experiments to simulate fire scenarios and obtain fire video data.

[0124] Below are some publicly available datasets of fire video images:

[0125] (1) Fire video dataset provided by Bilkent University in Turkey. This dataset is the most commonly used dataset by researchers of early fire video detection. It contains multiple videos of flames and smoke, as well as interference videos. However, most of these videos were recorded by early monitoring systems and have low resolution, only 240 pixels × 320 pixels.

[0126] (2) Fire video dataset released by the Computer Vision and Pattern Recognition Laboratory of Keimyung University in South Korea. The dataset contains videos of flames and smoke, and the scenes include both indoor and outdoor scenes.

[0127] (3) The fire database established by the Environmental Science Laboratory of the University of Corsica, France, was created for a vegetation fire modeling project. It primarily consists of images of vegetation flames, including visible light and near-infrared images. Account registration is required to download the data. The website also provides an upload function, allowing users to upload their own fire video image data.

[0128] (4) The forest fire database compiled by the Faculty of Electrical Engineering at the University of Split, Croatia, provides a very wide field of view of forest smoke videos.

[0129] (5) The forest fire database published by the Seville Microelectronics Institute in Spain was established for the project V-MOTE, which aims to use wireless sensor networks to build a video detection system and apply it to forest fire detection, building monitoring and border control.

[0130] (6) The fire database published by Professor Yuan Feiniu's research group at Jiangxi University of Finance and Economics. The website provides some smoke videos and interference videos, as well as four smoke image libraries and a video smoke detection demonstration program.

[0131] (7) Video fire data provided by the State Key Laboratory of Fire Science, University of Science and Technology of China. This dataset contains fire data used in relevant articles by the fire detection research group of the laboratory, including videos and pictures.

[0132] In addition, some research papers use fire video image data that has not yet been publicly available online. For example, Professor Zhou Zhiqiang's research group at Beijing Institute of Technology used high-quality forest smoke video in their paper. Due to the diversity of fire scenes and the significant differences in shooting conditions, it is not possible to simply compile all available fire video image data to create a database for all researchers in this field. Currently, researchers often only have a small amount of fire video image data available for their own research, achieving good recognition results with their own video image data. However, the lack of a unified database makes it difficult to reasonably evaluate the merits of algorithms and hinders the training of robust and highly generalizable video fire detection algorithms. Furthermore, because the invention targets the fusion detection of vivid flames across multiple time intervals, the above datasets are not suitable for the experimental requirements. Therefore, this experiment chose to collect data independently.

[0133] The data acquisition device used in this experiment was a Hikvision MV CE200 10GC industrial camera. The experimenters divided the dataset into indoor and outdoor sets: FIRE_IN and FIRE_OUT. FIRE_IN consisted of approximately 12,000 video frames, and FIRE_OUT consisted of approximately 18,000 video frames. Approximately 2,000 frames were subsequently filtered out due to image shift caused by camera movement. During shooting, the exposure gate was set to 15. Exposure time was adjusted to control the exposure. The camera supports a maximum exposure time of 700,000 µs. The initial exposure time was automatically selected by the camera based on the scene. Subsequent exposure time adjustments were made with a 150,000 µs difference to capture consecutive images at different exposure levels (script-controlled).

[0134] This experiment ultimately yielded two datasets, which were then used to compare the performance of different approaches. FIRE_IN represents indoor shooting, with differences in the ignition platform, shooting angle, and burning material; the overall environment is relatively simple. FIRE_OUT represents outdoor shooting, involving multi-scene acquisition and shooting in more complex scenarios.

[0135] (2.2.2) Experimental results and analysis.

[0136] Because this invention employs post-fusion detection—that is, the data is first fused into a single image before being fed into the detection network—the final detection result will also be presented in the fused image. To compare this with ordinary non-fused direct training detection, this experiment sets the final detection result to be mapped onto the original three images. In other words, the detection result in the fused image will be mapped onto the original three images. The following is a comparative experiment. The final test results are as follows:

[0137] Table 1 shows the performance test results of the ablation and replacement detection networks, while Table 2 shows the results using the same data and the same detection network (YOLOV5n) but different fusion methods, including traditional pixel-level methods and deep learning-based methods.

[0138] Table 1: Performance Comparison of the Invention and the Replacement Detection Network Scheme

[0139]

[0140] Table 2: Performance comparison of the fusion module in this invention with other fusion methods

[0141]

[0142] Analysis of the above experimental data reveals that the present invention achieves the best results compared to traditional methods and other image fusion methods when detecting indoor flame data, i.e., flames at medium to close range, and also has more outstanding performance in complex outdoor scenarios.

[0143] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A flame detection method based on flame specificity and multi-exposure image fusion, characterized in that, It includes: A flame recognition model is constructed, which includes the image fusion module EMEIF and the flame detection network; EM EIF is used according to n A fused image is generated from a series of consecutive multi-exposure images; a flame detection network is used to generate flame recognition results based on the fused image; the EMEIF consists of an encoder, a fusion unit, and a decoder; the encoder extracts features from the input multi-exposure images to obtain multiple feature maps F1~F2. n The fusion unit uses preset fusion weights W1~W1 based on relative exposure. n For F1 to F respectively n After weighting, channel stitching is performed to obtain fused features; the fused features are then processed by the decoder to obtain the fused image; the fusion weights W1~W n The generation method is as follows: S01: Calculate the feature map extracted from the original image at each exposure level using the following formula. F i Exposure value E i : ； In the above formula, I ( p ) represents the first element in the input image. p The normalized intensity value of each pixel; N This represents the total number of pixels in the image; S02: The exposure values ​​of each image are calculated using the following formula. E i Convert to relative exposure R i : ； S03: The relative exposure is calculated using the following non-linear function. R i Convert to dynamic weights W i : ； In the above formula, and To increase the two learnable weight coefficients in the fusion unit. This will significantly increase the weight of low-exposure images, while the weight of high-exposure images will approach 0; The threshold for relative exposure can be adjusted, affecting the average point of weight distribution; The initial model was constructed using EMEIF and a pre-trained flame detection network; it was then trained as follows: a loss function incorporating inter-image structural similarity was set. L SSIM and weight loss L wight loss function L 1. Train the initial model once, iteratively updating the weight coefficients of the fusion units in EMEIF; set the loss to include bounding box regression. L box Target confidence loss L obj Category classification loss Lcls joint losses L 2. Perform a second round of training on the model after the previous training; iteratively update the model parameters of the flame detection network; set parameters including... L 1. L box , L obj and L cls joint losses L 3. Perform three rounds of training on the model after the previous round of training, and iteratively update the model parameters of EMEIF and the flame detection network; Multiple consecutive multi-exposure images are input into a flame recognition model that has been trained for three rounds to achieve flame recognition.

2. The flame detection method based on flame specificity and multi-exposure image fusion according to claim 1, characterized in that: The flame detection network uses a pre-trained network based on YOLOv5; and / or The encoder in the image fusion module is a P-DenseBlock encoder; the P-DenseBlock encoder sequentially includes a first convolutional layer PConv with a 3×3 windmill-shaped convolutional module, and a second convolutional layer DPC1, a third convolutional layer DPC2, and a fourth convolutional layer DPC3 with a 3×3 depthwise separable convolutional kernel; the output of each convolutional layer in the P-DenseBlock encoder is residually concatenated with the outputs of subsequent convolutional layers; and / or The decoder in the image fusion module is a DenseNet decoder, which contains four convolutional layers with 3×3 kernels.

3. The flame detection method based on flame specificity and multi-exposure image fusion according to claim 1, characterized in that: One round of training uses a dataset consisting of real sample images; The second round of training uses a dataset composed of real sample images and fused images from the EMEIF output of the first round of training. The three rounds of training used a dataset consisting of real sample images and their augmented images.

4. The flame detection method based on flame specificity and multi-exposure image fusion according to claim 3, characterized in that, loss function L The expression for 1 is as follows: ； In the above formula, Indicates the input number of the first... i The original image at this exposure level; Represents a fused image; This is an evaluation function used to measure the structural similarity between the fused image and the input image; Indicates the first i Original image structure adjustment factor under various exposure levels; Used to calculate the square of the L2 normal form for each weight; Represents a parameter factor. =0.0001.

5. The flame detection method based on flame specificity and multi-exposure image fusion according to claim 4, characterized in that: loss function L The expression for 2 is as follows: L 2 =L box +L obj +L cls ； loss function L The expression for 3 is as follows: ； In the above formula, Represents the balance factor. =0.

1.

6. The flame detection method based on flame specificity and multi-exposure image fusion according to claim 1, characterized in that: The initial input of the flame recognition model is three consecutive multi-exposure images; during the inference phase, the three consecutive multi-exposure images are continuously acquired by the camera at a sampling interval of 0.2s, and the exposure of each image is adjusted by changing the exposure duration.

7. A computer program product comprising a computer program, characterized in that: When the computer program is executed by the processor, it creates a trained flame recognition model in the flame detection method based on flame specificity and multi-exposure image fusion as described in any one of claims 1-6, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

8. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it creates a trained flame recognition model in the flame detection method based on flame specificity and multi-exposure image fusion as described in any one of claims 1-6, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the computer program is executed by the processor, it creates a trained flame recognition model in the flame detection method based on flame specificity and multi-exposure image fusion as described in any one of claims 1-6, thereby enabling the identification of flames contained in multiple consecutive input multi-exposure images.

Images